sfeed

simple feed reader - forked from git.codemadness.org/sfeed
git clone git://src.gearsix.net/sfeedsfeed.zip
Log | Files | Refs | Atom | README | LICENSE

sfeedrc.5 (raw) (6671B)


   1 .Dd February 9, 2025
   2 .Dt SFEEDRC 5
   3 .Os
   4 .Sh NAME
   5 .Nm sfeedrc
   6 .Nd sfeed_update(1) configuration file
   7 .Sh DESCRIPTION
   8 .Nm
   9 is the configuration file for
  10 .Xr sfeed_update 1
  11 and is evaluated as a shellscript.
  12 .Sh VARIABLES
  13 .Bl -tag -width Ds
  14 .It Va sfeedpath
  15 can be set for the directory to store the TAB-separated feed files.
  16 The default is
  17 .Pa $HOME/.sfeed/feeds .
  18 .It Va maxjobs
  19 can be used to change the amount of concurrent
  20 .Fn feed
  21 jobs.
  22 The default is 16.
  23 .El
  24 .Sh FUNCTIONS
  25 .Bl -tag -width Ds
  26 .It Fn feeds
  27 This function is the required "main" entry-point function called from
  28 .Xr sfeed_update 1 .
  29 .It Fn feed "name" "feedurl" "basesiteurl" "encoding"
  30 Inside the
  31 .Fn feeds
  32 function feeds can be defined by calling the
  33 .Fn feed
  34 function.
  35 Its arguments are:
  36 .Bl -tag -width Ds
  37 .It Fa name
  38 Name of the feed, this is also used as the filename for the TAB-separated
  39 feed file.
  40 The feed name cannot contain the '/' character because it is a path separator,
  41 they will be replaced with '_'.
  42 Each
  43 .Fa name
  44 should be unique.
  45 .It Fa feedurl
  46 URL to fetch the RSS/Atom data from.
  47 This is usually a HTTP or HTTPS URL.
  48 .It Op Fa basesiteurl
  49 Base URL of the feed links.
  50 This argument allows fixing relative item links.
  51 .Pp
  52 According to the RSS and Atom specification, feeds should always have absolute
  53 URLs, but this is not always the case in practice.
  54 .It Op Fa encoding
  55 Feeds are converted from this
  56 .Ar encoding
  57 to UTF-8.
  58 The
  59 .Ar encoding
  60 should be a usable
  61 character-set name for the
  62 .Xr iconv 1
  63 tool.
  64 .El
  65 .El
  66 .Sh OVERRIDE FUNCTIONS
  67 Because
  68 .Xr sfeed_update 1
  69 is a shellscript each function can be overridden to change its behaviour.
  70 Notable functions are:
  71 .Bl -tag -width Ds
  72 .It Fn fetch "name" "url" "feedfile"
  73 Fetch feed from URL and write the data to stdout.
  74 Its arguments are:
  75 .Bl -tag -width Ds
  76 .It Fa name
  77 Feed name.
  78 .It Fa url
  79 URL to fetch.
  80 .It Fa feedfile
  81 Used feedfile (useful for comparing modification times).
  82 .El
  83 .Pp
  84 By default the tool
  85 .Xr curl 1
  86 is used.
  87 .It Fn convertencoding "name" "from" "to"
  88 Convert data from stdin from one text-encoding to another and write it to
  89 stdout.
  90 Its arguments are:
  91 .Bl -tag -width Ds
  92 .It Fa name
  93 Feed name.
  94 .It Fa from
  95 From text-encoding.
  96 .It Fa to
  97 To text-encoding.
  98 .El
  99 .Pp
 100 By default the tool
 101 .Xr iconv 1
 102 is used.
 103 .It Fn parse "name" "feedurl" "basesiteurl"
 104 Read RSS/Atom XML data from stdin, convert and write it as
 105 .Xr sfeed 5
 106 data to stdout.
 107 Its arguments are:
 108 .Bl -tag -width Ds
 109 .It Fa name
 110 Feed name.
 111 .It Fa feedurl
 112 URL of the feed.
 113 .It Fa basesiteurl
 114 Base URL of the feed links.
 115 This argument allows to fix relative item links.
 116 .El
 117 .It Fn filter "name" "url"
 118 Filter
 119 .Xr sfeed 5
 120 data from stdin and write it to stdout.
 121 Its arguments are:
 122 .Bl -tag -width Ds
 123 .It Fa name
 124 Feed name.
 125 .It Fa url
 126 URL of the feed.
 127 .El
 128 .It Fn merge "name" "oldfile" "newfile"
 129 Merge
 130 .Xr sfeed 5
 131 data of oldfile with newfile and write it to stdout.
 132 Its arguments are:
 133 .Bl -tag -width Ds
 134 .It Fa name
 135 Feed name.
 136 .It Fa oldfile
 137 Old file.
 138 .It Fa newfile
 139 New file.
 140 .El
 141 .It Fn order "name" "url"
 142 Sort
 143 .Xr sfeed 5
 144 data from stdin and write it to stdout.
 145 Its arguments are:
 146 .Bl -tag -width Ds
 147 .It Fa name
 148 Feed name.
 149 .It Fa url
 150 URL of the feed.
 151 .El
 152 .El
 153 .Sh EXAMPLES
 154 An example configuration file is included named sfeedrc.example and also
 155 shown below:
 156 .Bd -literal -offset 4n
 157 #sfeedpath="$HOME/.sfeed/feeds"
 158 
 159 # list of feeds to fetch:
 160 feeds() {
 161 	# feed <name> <feedurl> [basesiteurl] [encoding]
 162 	feed "codemadness" "https://www.codemadness.org/atom_content.xml"
 163 	feed "explosm" "http://feeds.feedburner.com/Explosm"
 164 	feed "golang github releases" "https://github.com/golang/go/releases.atom"
 165 	feed "linux kernel" "https://www.kernel.org/feeds/kdist.xml" "https://www.kernel.org"
 166 	feed "reddit openbsd" "https://old.reddit.com/r/openbsd/.rss"
 167 	feed "slashdot" "http://rss.slashdot.org/Slashdot/slashdot" "http://slashdot.org"
 168 	feed "tweakers" "http://feeds.feedburner.com/tweakers/mixed" "http://tweakers.net" "iso-8859-1"
 169 	# get youtube Atom feed: curl -s -L 'https://www.youtube.com/user/gocoding/videos' | sfeed_web | cut -f 1
 170 	feed "youtube golang" "https://www.youtube.com/feeds/videos.xml?channel_id=UCO3LEtymiLrgvpb59cNsb8A"
 171 	feed "xkcd" "https://xkcd.com/atom.xml" "https://xkcd.com"
 172 }
 173 .Ed
 174 .Pp
 175 To change the default
 176 .Xr curl 1
 177 options for fetching the data, the
 178 .Fn fetch
 179 function can be overridden and added at the top of the
 180 .Nm
 181 file, for example:
 182 .Bd -literal -offset 4n
 183 # fetch(name, url, feedfile)
 184 fetch() {
 185 	# allow for 1 redirect, set User-Agent, timeout is 15 seconds.
 186 	curl -L --max-redirs 1 -H "User-Agent: 007" -f -s -m 15 \e
 187 		"$2" 2>/dev/null
 188 }
 189 .Ed
 190 .Pp
 191 Caching, incremental data updates and bandwidth saving
 192 .Pp
 193 For HTTP servers that support it some bandwidth saving can be done by changing
 194 some of the default curl options.
 195 These options can come at a cost of some privacy, because it exposes additional
 196 metadata from the previous request.
 197 .Pp
 198 .Bl -bullet -compact
 199 .It
 200 The curl ETag options (--etag-save and --etag-compare) can be used to store and
 201 send the previous ETag header value.
 202 curl version 7.73+ is recommended for it to work properly.
 203 .It
 204 The curl -z option can be used to send the modification date of a local file as
 205 a HTTP If-Modified-Since request header.
 206 The server can then respond if the data is modified or not or respond with only
 207 the incremental data.
 208 .It
 209 The curl --compressed option can be used to indicate the client supports
 210 decompression.
 211 Because RSS/Atom feeds are textual XML data this generally compresses very
 212 well.
 213 .It
 214 The example below also sets the User-Agent to sfeed, because some CDNs block
 215 HTTP clients based on the User-Agent request header.
 216 .El
 217 .Pp
 218 Example:
 219 .Bd -literal -offset 4n
 220 mkdir -p "$HOME/.sfeed/etags" "$HOME/.sfeed/lastmod"
 221 
 222 # fetch(name, url, feedfile)
 223 fetch() {
 224 	basename="$(basename "$3")"
 225 	etag="$HOME/.sfeed/etags/${basename}"
 226 	lastmod="$HOME/.sfeed/lastmod/${basename}"
 227 	output="${sfeedtmpdir}/feeds/${filename}.xml"
 228 
 229 	curl \e
 230 		-f -s -m 15 \e
 231 		-L --max-redirs 0 \e
 232 		-H "User-Agent: sfeed" \e
 233 		--compressed \e
 234 		--etag-save "${etag}" --etag-compare "${etag}" \e
 235 		-R -o "${output}" \e
 236 		-z "${lastmod}" \e
 237 		"$2" 2>/dev/null || return 1
 238 
 239 	# succesful, but no file written: assume it is OK and Not Modified.
 240 	[ -e "${output}" ] || return 0
 241 
 242 	# use server timestamp from curl -R to set Last-Modified.
 243 	touch -r "${output}" "${lastmod}" 2>/dev/null
 244 	cat "${output}" 2>/dev/null
 245 	# use write output status, other errors are ignored here.
 246 	fetchstatus="$?"
 247 	rm -f "${output}" 2>/dev/null
 248 	return "${fetchstatus}"
 249 }
 250 .Ed
 251 .Pp
 252 The README file has more examples.
 253 .Sh SEE ALSO
 254 .Xr curl 1 ,
 255 .Xr iconv 1 ,
 256 .Xr sfeed_update 1 ,
 257 .Xr sh 1
 258 .Sh AUTHORS
 259 .An Hiltjo Posthuma Aq Mt hiltjo@codemadness.org