FeedMixer
FeedMixerisalittlewebservice(Pytho3/WSGI)whichtakesalistoffeedURLsadcombiesthemitoasigle(Atom,RSS,orJSON)feed.Usefulforpersoalewsaggregators,"plaet"-likewebsites,etc.
StatusChagelogv2.3.1Morecosistetbuilds:updatedepedeciesiPipfile.lock(whichalsoseemstoworkbetterwithewerversiosofpipev)adpiDockerfilebaseimagetospecifichashv2.3.0Replaceo-diskcachewithi-memorycache.Thissimplifiesapplicatiocodeadadmiistratio(do'thavetoworryaboutpruigthecachedatabase)v2.2.0FixhadligofRSSfeedswithmissigpubDatessothattheysorttothebottomisteadofthrowigaexceptiodurigsortigv2.1.0FixhadligofRSSeclosuresadAtomlikssothattheyareicludedioutput(importatifyou'retryigtoaggregatepodcastsorsimilar)v2.0.0TheJSONoutputowcoformstoJSONFeedversio1.Thisbreaksayclietwhichdepedsothepreviousad-hocJSONformat.Thatlegacyformatwillcotiuetobemaitaiedithev1brach,soayclietswhichdo'twattoupdatetotheJSONFeedformatshoulddepedothatbrach.v1.0.0StableAPI.I'musigitiproductioforsmallpersoal"plaet"-likefeedaggregators.APIFeedMixerexposesthreeedpoits:
/atom/rss/jsoWhesetaGETrequesttheyreturaAtom,aRSS2.0,oraJSONfeed,respectively.ThequerystrigoftheGETrequestcacotaithesefields:
fAurl-ecodedURLofafeed(ayversioofAtomorRSS).Toicludemultiplefeeds,simplyicludemultipleffields.Theumberofetriestokeepfromeachfield(pass0tokeepalletries,whichisthedefaultifofieldisprovided).fullIfsettoaythig,preferthefulletrycotet;ifabset,prefertheshorteretrysummary.FeaturesAPICombieseveralfeeds(justaboutayversioofAtomadRSSshouldwork)itoasiglefeedOptioallyreturolythemostrecetitemsfromeachiputfeedCotrolwhethertheoutputfeedcotaisolythesummaryortheetirecotetoftheiputfeeditemsParserresultsarememoizedsothatrepeatedrequestsforthesamefeedcabereturedwithoutre-parsig.IcludedWSGIappTheprovidedfeedmixer_wsgi.pyapplicatiousesasessiothatcachesHTTPresposessothatrepeatedlyfetchigthesamesetsoffeedscausuallyberespodedtoquicklybytheFeedMixerservice.
TheFeedMixerobjectcabepassedacustomrequests.sessioobjectusedtomakeHTTPrequests,whichallowsflexiblecustomizatioihowrequestsaremadeifyoueedthat.
No-featuresFeedMixerdoesot(yet?)doayresourcerestrictioitself:
AuthorizatioRatelimitigCORSrestrictioTOprotectyouristallatioeithercofigureafrot-edhttpproxytotakecareofyourrequiredrestrictios(Ngixisagoodchoice),or/adusesuitableWSGImiddleware.
IstallatioCloethisrepository:$gitcloehttps://github.com/cristoper/feedmixer.git$cdfeedmixerRecommeded:usepipevtocreateavirtualevadistalldepedecies:$pipev--threesycTheprojectcosistsofthreemodules:
feedmixer.py-cotaisthecorelogicfeedmixer_api.py-cotaistheFalco-basedAPI.Callwsgi_app()togetaWSGI-compliatobjecttohost.feedmixer_wsgi.py-cotaisaactualWSGIapplicatiowhichcabeusedas-isorasastartigpoittocreateyourowcustomFeedMixerservice.RuLocallyThefeedmixer_wsgimoduleistatiatesthefeedmixerWSGIobject(withsesibledefaultsadarotatiglogfile)asbothapiadapplicatio(defaultamesusedbycommoWSGIservers).Tostarttheservicewithguicor,forexample,cloetherepositoryaditherootdirectoryru:
$pipevrupip3istallguicor$pipevruguicorfeedmixer_wsgiNotethatthetop-levelistalldirectorymustbewritablebytheserverruigtheapp,becauseitcreatesthelogfiles('fm.log'ad'fm.log.1')there.
Asaexample,assumigaistaceoftheFeedMixerappisruigothelocalhostoport8000,let'sfetchtheewestetryeachfromthefollowigAtomadRSSfeeds:
https://catswhisker.xyz/shaarli/?do=atomhttps://hrss.org/ewestThecostructedURLtoGETis:
https://localhost:8000/atom?f=https://catswhisker.xyz/shaarli/?do=atom&f=https://hrss.org/ewest&=1
EterigititoabrowserwillreturaAtomfeedwithtwoetries.ToGETitfromaclietprogramatically,remembertoURL-ecodetheffields:
$curl'localhost:8000/atom?f=https%3A%2F%2Fcatswhisker.xyz%2Fshaarli%2F%3Fdo%3Datom&f=https%3A%2F%2Fhrss.org%2Fewest&=1'HTTPieisaicecommad-liehttpclietthatmakestestigRESTfulservicesmorepleasat:
$pip3istallhttpie$httplocalhost:8000/jsof==https://hrss.org/ewestf==https://catswhisker.xyz/atom.xml==1YoushouldseesomeJSONFeedoutput(sicewearerequestigfromthe/jsoedpoit):
HTTP/1.1200OKCoectio:closeDate:Thu,23Ja202003:53:45GMTServer:guicor/20.0.4cotet-legth:1296cotet-type:applicatio/jso{"versio":"https://jsofeed.org/versio/1","title":"FeedMixerfeed","home_page_url":"https://localhost:8000/jso?f=http%3A%2F%2Fhrss.org%2Fewest&f=https%3A%2F%2Fcatswhisker.xyz%2Fatom.xml&=1","descriptio":"jsofeedcreatedbyFeedMixer.","items":[{"title":"KyrsteSiema,theOlyAti-NetNeutralityDem,LikedtoComcastSuperPac","cotet_html":"<p>ArticleURL:<ahref=\"https://prospect.org/politics/kyrste-siema-ati-et-eutrality-super-pac-comcast-lobbyist/\">https://prospect.org/politics/kyrste-siema-ati-et-eutrality-super-pac-comcast-lobbyist/</a></p>\<p>CommetsURL:<ahref=\"https://ews.ycombiator.com/item?id=22124592\">https://ews.ycombiator.com/item?id=22124592</a></p>\<p>Poits:1</p>\<p>#Commets:0</p>","url":"https://prospect.org/politics/kyrste-siema-ati-et-eutrality-super-pac-comcast-lobbyist/","id":"https://ews.ycombiator.com/item?id=22124592","author":{"ame":"joeyespo"},"date_published":"2020-01-23T03:32:19Z","date_modified":"2020-01-23T03:32:19Z"},{"title":"FORoudupDecember2019","cotet_html":"I'vestartedkittigagai.","url":"https://catswhisker.xyz/log/2019/12/3/fo_december/","id":"tag:catswhisker.xyz,2019-12-04:/log/2019/12/3/fo_december/","author":{"ame":"A.Cyic","url":"https://catswhisker.xyz/about/"},"date_published":"2019-12-04T04:48:59Z","date_modified":"2019-12-04T04:48:59Z"}]}DeployDeployFeedMixerusigayWSGI-compliatserver(uswgi,guicor,mod_wsgi,...).Foraproductiodeploymet,putaasychrooushttpproxy(likeNgix)ifrotofFeedMixertoprotectitfromtoomayadslowcoectios(aswellastoprovideSSLtermiatio,additioalcachig,authoriziatio,etc.,asrequired)
Refertothedocumetatiooftheserverofyourchoice.
ApacheForotesodeployigbehidApache,seeapache.rst(fromhtmldocs:apache.html)
DockerAalterativetousigavirtualevforbothbuildigaddeployigistoruFeedMixeriaDockercotaier.TheicludedDockerfilewillproduceaimagewhichrusFeedMixerusigguicor.
Buildtheimagefromthefeedmixerdirectory:
$dockerbuild.-tfeedmixerRuititheforegroud:
$dockerru-p8000:8000feedmixerNowfromaothertermialyoushouldbeabletocoecttoFeedMixerolocalhostport8000justasitheexampleabove.
TroubleshootigUsigtheprovidedfeedmixer_wsgi.pyapplicatio,iformatioaderrorsareloggedtothefilefm.logithedirectorytheapplicatioisstartedfrom(autorotatedwithasigleoldlogcalledfm.1.log).
AyerrorsecouteredifetchigadparsigremotefeedsarereportediacustomHTTPheadercalledX-fm-errors.
HackigFirstistallasperistructiosabove.
DocumetatioOtherthathisREADME,thedocumetatioisithedocstrigs.Tobuildaprettyversio(HTML)usigSphix:
IstallSphixdepedecies:$pipevrupipistall-rdoc/requiremets.txtChagetodoc/directory:$cddocBuild:$pipevrumakehtmlView:$x-www-browser_build/html/idex.htmlTestsTestsareithetestdirectoryadPythowillfidadruthemwith:
$pipevrupytho3-muittestTypecheckigTochecktypesusigmypy:
$MYPYPATH=stub/mypy--igore-missig-imports-pfeedmixerNoteverythigisstubbedout,butcabeusefulforcatchigbugsafterchagigfeedparser.py
GethelpFeelfreetoopeaissueoGithubforhelp:https://github.com/cristoper/feedmixer/issues
SupporttheprojectIfthispackagewasusefultoyou,pleasecosidersupportigmyworkothisadotherope-sourceprojectsbymakigasmall(likeatip)oe-timedoatio:doateviaPayPal
Ifyou'relookigtocotractaPythodeveloper,Imightbeabletohelp.Cotactmeatchris.burkhardt@orageoiseproductio.com
LiceseTheprojectislicesedudertheWTFPLlicese,withoutwarratyofaykid.
评论