FeedMixerisalittlewebservice(Python3/WSGI)whichtakesalistoffeedURLsandcombinesthemintoasingle(Atom,RSS,orJSON)feed.Usefulforpersonalnewsaggregators,"planet"-likewebsites,etc.
StatusChangelogv2.3.1Moreconsistentbuilds:updatedependenciesinPipfile.lock(whichalsoseemstoworkbetterwithnewerversionsofpipenv)andpinDockerfilebaseimagetospecifichashv2.3.0Replaceon-diskcachewithin-memorycache.Thissimplifiesapplicationcodeandadministration(don'thavetoworryaboutpruningthecachedatabase)v2.2.0FixhandlingofRSSfeedswithmissingpubDatessothattheysorttothebottominsteadofthrowinganexceptionduringsortingv2.1.0FixhandlingofRSSenclosuresandAtomlinkssothattheyareincludedinoutput(importantifyou'retryingtoaggregatepodcastsorsimilar)v2.0.0TheJSONoutputnowconformstoJSONFeedversion1.Thisbreaksanyclientwhichdependsonthepreviousad-hocJSONformat.Thatlegacyformatwillcontinuetobemaintainedinthev1branch,soanyclientswhichdon'twanttoupdatetotheJSONFeedformatshoulddependonthatbranch.v1.0.0StableAPI.I'musingitinproductionforsmallpersonal"planet"-likefeedaggregators.APIFeedMixerexposesthreeendpoints:
/atom/rss/jsonWhensentaGETrequesttheyreturnanAtom,anRSS2.0,oraJSONfeed,respectively.ThequerystringoftheGETrequestcancontainthesefields:
fAurl-encodedURLofafeed(anyversionofAtomorRSS).Toincludemultiplefeeds,simplyincludemultipleffields.nThenumberofentriestokeepfromeachfield(pass0tokeepallentries,whichisthedefaultifnonfieldisprovided).fullIfsettoanything,preferthefullentrycontent;ifabsent,prefertheshorterentrysummary.FeaturesAPICombineseveralfeeds(justaboutanyversionofAtomandRSSshouldwork)intoasinglefeedOptionallyreturnonlythenmostrecentitemsfromeachinputfeedControlwhethertheoutputfeedcontainsonlythesummaryortheentirecontentoftheinputfeeditemsParserresultsarememoizedsothatrepeatedrequestsforthesamefeedcanbereturnedwithoutre-parsing.IncludedWSGIappTheprovidedfeedmixer_wsgi.pyapplicationusesasessionthatcachesHTTPresponsessothatrepeatedlyfetchingthesamesetsoffeedscanusuallyberespondedtoquicklybytheFeedMixerservice.
TheFeedMixerobjectcanbepassedacustomrequests.sessionobjectusedtomakeHTTPrequests,whichallowsflexiblecustomizationinhowrequestsaremadeifyouneedthat.
Non-featuresFeedMixerdoesnot(yet?)doanyresourcerestrictionitself:
AuthorizationRatelimitingCORSrestrictionTOprotectyourinstallationeitherconfigureafront-endhttpproxytotakecareofyourrequiredrestrictions(Nginxisagoodchoice),or/andusesuitableWSGImiddleware.
InstallationClonethisrepository:$gitclonehttps://github.com/cristoper/feedmixer.git$cdfeedmixerRecommended:usepipenvtocreateavirtualenvandinstalldependencies:$pipenv--threesyncTheprojectconsistsofthreemodules:
feedmixer.py-containsthecorelogicfeedmixer_api.py-containstheFalcon-basedAPI.Callwsgi_app()togetaWSGI-compliantobjecttohost.feedmixer_wsgi.py-containsanactualWSGIapplicationwhichcanbeusedas-isorasastartingpointtocreateyourowncustomFeedMixerservice.RunLocallyThefeedmixer_wsgimoduleinstantiatesthefeedmixerWSGIobject(withsensibledefaultsandarotatinglogfile)asbothapiandapplication(defaultnamesusedbycommonWSGIservers).Tostarttheservicewithgunicorn,forexample,clonetherepositoryandintherootdirectoryrun:
$pipenvrunpip3installgunicorn$pipenvrungunicornfeedmixer_wsgiNotethatthetop-levelinstalldirectorymustbewritablebytheserverrunningtheapp,becauseitcreatesthelogfiles('fm.log'and'fm.log.1')there.
Asanexample,assuminganinstanceoftheFeedMixerappisrunningonthelocalhostonport8000,let'sfetchthenewestentryeachfromthefollowingAtomandRSSfeeds:
https://catswhisker.xyz/shaarli/?do=atomhttps://hnrss.org/newestTheconstructedURLtoGETis:
https://localhost:8000/atom?f=https://catswhisker.xyz/shaarli/?do=atom&f=https://hnrss.org/newest&n=1
EnteringitintoabrowserwillreturnanAtomfeedwithtwoentries.ToGETitfromaclientprogramatically,remembertoURL-encodetheffields:
$curl'localhost:8000/atom?f=https%3A%2F%2Fcatswhisker.xyz%2Fshaarli%2F%3Fdo%3Datom&f=https%3A%2F%2Fhnrss.org%2Fnewest&n=1'HTTPieisanicecommand-linehttpclientthatmakestestingRESTfulservicesmorepleasant:
$pip3installhttpie$httplocalhost:8000/jsonf==https://hnrss.org/newestf==https://catswhisker.xyz/atom.xmln==1YoushouldseesomeJSONFeedoutput(sincewearerequestingfromthe/jsonendpoint):
HTTP/1.1200OKConnection:closeDate:Thu,23Jan202003:53:45GMTServer:gunicorn/20.0.4content-length:1296content-type:application/json{"version":"https://jsonfeed.org/version/1","title":"FeedMixerfeed","home_page_url":"https://localhost:8000/json?f=http%3A%2F%2Fhnrss.org%2Fnewest&f=https%3A%2F%2Fcatswhisker.xyz%2Fatom.xml&n=1","description":"jsonfeedcreatedbyFeedMixer.","items":[{"title":"KyrstenSinema,theOnlyAnti-NetNeutralityDem,LinkedtoComcastSuperPac","content_html":"<p>ArticleURL:<ahref=\"https://prospect.org/politics/kyrsten-sinema-anti-net-neutrality-super-pac-comcast-lobbyist/\">https://prospect.org/politics/kyrsten-sinema-anti-net-neutrality-super-pac-comcast-lobbyist/</a></p>\n<p>CommentsURL:<ahref=\"https://news.ycombinator.com/item?id=22124592\">https://news.ycombinator.com/item?id=22124592</a></p>\n<p>Points:1</p>\n<p>#Comments:0</p>","url":"https://prospect.org/politics/kyrsten-sinema-anti-net-neutrality-super-pac-comcast-lobbyist/","id":"https://news.ycombinator.com/item?id=22124592","author":{"name":"joeyespo"},"date_published":"2020-01-23T03:32:19Z","date_modified":"2020-01-23T03:32:19Z"},{"title":"FORoundupDecember2019","content_html":"I'vestartedknittingagain.","url":"https://catswhisker.xyz/log/2019/12/3/fo_december/","id":"tag:catswhisker.xyz,2019-12-04:/log/2019/12/3/fo_december/","author":{"name":"A.Cynic","url":"https://catswhisker.xyz/about/"},"date_published":"2019-12-04T04:48:59Z","date_modified":"2019-12-04T04:48:59Z"}]}DeployDeployFeedMixerusinganyWSGI-compliantserver(uswgi,gunicorn,mod_wsgi,...).Foraproductiondeployment,putanasynchronoushttpproxy(likeNginx)infrontofFeedMixertoprotectitfromtoomanyandslowconnections(aswellastoprovideSSLtermination,additionalcaching,authoriziation,etc.,asrequired)
Refertothedocumentationoftheserverofyourchoice.
ApacheFornotesondeployingbehindApache,seeapache.rst(fromhtmldocs:apache.html)
DockerAnalternativetousingavirtualenvforbothbuildinganddeployingistorunFeedMixerinaDockercontainer.TheincludedDockerfilewillproduceanimagewhichrunsFeedMixerusinggunicorn.
Buildtheimagefromthefeedmixerdirectory:
$dockerbuild.-tfeedmixerRunitintheforeground:
$dockerrun-p8000:8000feedmixerNowfromanotherterminalyoushouldbeabletoconnecttoFeedMixeronlocalhostport8000justasintheexampleabove.
TroubleshootingUsingtheprovidedfeedmixer_wsgi.pyapplication,informationanderrorsareloggedtothefilefm.loginthedirectorytheapplicationisstartedfrom(autorotatedwithasingleoldlogcalledfm.1.log).
AnyerrorsencounteredinfetchingandparsingremotefeedsarereportedinacustomHTTPheadercalledX-fm-errors.
HackingFirstinstallasperinstructionsabove.
DocumentationOtherthanthisREADME,thedocumentationisinthedocstrings.Tobuildaprettyversion(HTML)usingSphinx:
InstallSphinxdependencies:$pipenvrunpipinstall-rdoc/requirements.txtChangetodoc/directory:$cddocBuild:$pipenvrunmakehtmlView:$x-www-browser_build/html/index.htmlTestsTestsareinthetestdirectoryandPythonwillfindandrunthemwith:
$pipenvrunpython3-munittestTypecheckingTochecktypesusingmypy:
$MYPYPATH=stub/mypy--ignore-missing-imports-pfeedmixerNoteverythingisstubbedout,butcanbeusefulforcatchingbugsafterchangingfeedparser.py
GethelpFeelfreetoopenanissueonGithubforhelp:https://github.com/cristoper/feedmixer/issues
SupporttheprojectIfthispackagewasusefultoyou,pleaseconsidersupportingmyworkonthisandotheropen-sourceprojectsbymakingasmall(likeatip)one-timedonation:donateviaPayPal
Ifyou'relookingtocontractaPythondeveloper,Imightbeabletohelp.Contactmeatchris.burkhardt@orangenoiseproduction.com
LicenseTheprojectislicensedundertheWTFPLlicense,withoutwarrantyofanykind.
评论