sfeedrc.5 (raw) (6671B)
1 .Dd February 9, 2025 2 .Dt SFEEDRC 5 3 .Os 4 .Sh NAME 5 .Nm sfeedrc 6 .Nd sfeed_update(1) configuration file 7 .Sh DESCRIPTION 8 .Nm 9 is the configuration file for 10 .Xr sfeed_update 1 11 and is evaluated as a shellscript. 12 .Sh VARIABLES 13 .Bl -tag -width Ds 14 .It Va sfeedpath 15 can be set for the directory to store the TAB-separated feed files. 16 The default is 17 .Pa $HOME/.sfeed/feeds . 18 .It Va maxjobs 19 can be used to change the amount of concurrent 20 .Fn feed 21 jobs. 22 The default is 16. 23 .El 24 .Sh FUNCTIONS 25 .Bl -tag -width Ds 26 .It Fn feeds 27 This function is the required "main" entry-point function called from 28 .Xr sfeed_update 1 . 29 .It Fn feed "name" "feedurl" "basesiteurl" "encoding" 30 Inside the 31 .Fn feeds 32 function feeds can be defined by calling the 33 .Fn feed 34 function. 35 Its arguments are: 36 .Bl -tag -width Ds 37 .It Fa name 38 Name of the feed, this is also used as the filename for the TAB-separated 39 feed file. 40 The feed name cannot contain the '/' character because it is a path separator, 41 they will be replaced with '_'. 42 Each 43 .Fa name 44 should be unique. 45 .It Fa feedurl 46 URL to fetch the RSS/Atom data from. 47 This is usually a HTTP or HTTPS URL. 48 .It Op Fa basesiteurl 49 Base URL of the feed links. 50 This argument allows fixing relative item links. 51 .Pp 52 According to the RSS and Atom specification, feeds should always have absolute 53 URLs, but this is not always the case in practice. 54 .It Op Fa encoding 55 Feeds are converted from this 56 .Ar encoding 57 to UTF-8. 58 The 59 .Ar encoding 60 should be a usable 61 character-set name for the 62 .Xr iconv 1 63 tool. 64 .El 65 .El 66 .Sh OVERRIDE FUNCTIONS 67 Because 68 .Xr sfeed_update 1 69 is a shellscript each function can be overridden to change its behaviour. 70 Notable functions are: 71 .Bl -tag -width Ds 72 .It Fn fetch "name" "url" "feedfile" 73 Fetch feed from URL and write the data to stdout. 74 Its arguments are: 75 .Bl -tag -width Ds 76 .It Fa name 77 Feed name. 78 .It Fa url 79 URL to fetch. 80 .It Fa feedfile 81 Used feedfile (useful for comparing modification times). 82 .El 83 .Pp 84 By default the tool 85 .Xr curl 1 86 is used. 87 .It Fn convertencoding "name" "from" "to" 88 Convert data from stdin from one text-encoding to another and write it to 89 stdout. 90 Its arguments are: 91 .Bl -tag -width Ds 92 .It Fa name 93 Feed name. 94 .It Fa from 95 From text-encoding. 96 .It Fa to 97 To text-encoding. 98 .El 99 .Pp 100 By default the tool 101 .Xr iconv 1 102 is used. 103 .It Fn parse "name" "feedurl" "basesiteurl" 104 Read RSS/Atom XML data from stdin, convert and write it as 105 .Xr sfeed 5 106 data to stdout. 107 Its arguments are: 108 .Bl -tag -width Ds 109 .It Fa name 110 Feed name. 111 .It Fa feedurl 112 URL of the feed. 113 .It Fa basesiteurl 114 Base URL of the feed links. 115 This argument allows to fix relative item links. 116 .El 117 .It Fn filter "name" "url" 118 Filter 119 .Xr sfeed 5 120 data from stdin and write it to stdout. 121 Its arguments are: 122 .Bl -tag -width Ds 123 .It Fa name 124 Feed name. 125 .It Fa url 126 URL of the feed. 127 .El 128 .It Fn merge "name" "oldfile" "newfile" 129 Merge 130 .Xr sfeed 5 131 data of oldfile with newfile and write it to stdout. 132 Its arguments are: 133 .Bl -tag -width Ds 134 .It Fa name 135 Feed name. 136 .It Fa oldfile 137 Old file. 138 .It Fa newfile 139 New file. 140 .El 141 .It Fn order "name" "url" 142 Sort 143 .Xr sfeed 5 144 data from stdin and write it to stdout. 145 Its arguments are: 146 .Bl -tag -width Ds 147 .It Fa name 148 Feed name. 149 .It Fa url 150 URL of the feed. 151 .El 152 .El 153 .Sh EXAMPLES 154 An example configuration file is included named sfeedrc.example and also 155 shown below: 156 .Bd -literal -offset 4n 157 #sfeedpath="$HOME/.sfeed/feeds" 158 159 # list of feeds to fetch: 160 feeds() { 161 # feed <name> <feedurl> [basesiteurl] [encoding] 162 feed "codemadness" "https://www.codemadness.org/atom_content.xml" 163 feed "explosm" "http://feeds.feedburner.com/Explosm" 164 feed "golang github releases" "https://github.com/golang/go/releases.atom" 165 feed "linux kernel" "https://www.kernel.org/feeds/kdist.xml" "https://www.kernel.org" 166 feed "reddit openbsd" "https://old.reddit.com/r/openbsd/.rss" 167 feed "slashdot" "http://rss.slashdot.org/Slashdot/slashdot" "http://slashdot.org" 168 feed "tweakers" "http://feeds.feedburner.com/tweakers/mixed" "http://tweakers.net" "iso-8859-1" 169 # get youtube Atom feed: curl -s -L 'https://www.youtube.com/user/gocoding/videos' | sfeed_web | cut -f 1 170 feed "youtube golang" "https://www.youtube.com/feeds/videos.xml?channel_id=UCO3LEtymiLrgvpb59cNsb8A" 171 feed "xkcd" "https://xkcd.com/atom.xml" "https://xkcd.com" 172 } 173 .Ed 174 .Pp 175 To change the default 176 .Xr curl 1 177 options for fetching the data, the 178 .Fn fetch 179 function can be overridden and added at the top of the 180 .Nm 181 file, for example: 182 .Bd -literal -offset 4n 183 # fetch(name, url, feedfile) 184 fetch() { 185 # allow for 1 redirect, set User-Agent, timeout is 15 seconds. 186 curl -L --max-redirs 1 -H "User-Agent: 007" -f -s -m 15 \e 187 "$2" 2>/dev/null 188 } 189 .Ed 190 .Pp 191 Caching, incremental data updates and bandwidth saving 192 .Pp 193 For HTTP servers that support it some bandwidth saving can be done by changing 194 some of the default curl options. 195 These options can come at a cost of some privacy, because it exposes additional 196 metadata from the previous request. 197 .Pp 198 .Bl -bullet -compact 199 .It 200 The curl ETag options (--etag-save and --etag-compare) can be used to store and 201 send the previous ETag header value. 202 curl version 7.73+ is recommended for it to work properly. 203 .It 204 The curl -z option can be used to send the modification date of a local file as 205 a HTTP If-Modified-Since request header. 206 The server can then respond if the data is modified or not or respond with only 207 the incremental data. 208 .It 209 The curl --compressed option can be used to indicate the client supports 210 decompression. 211 Because RSS/Atom feeds are textual XML data this generally compresses very 212 well. 213 .It 214 The example below also sets the User-Agent to sfeed, because some CDNs block 215 HTTP clients based on the User-Agent request header. 216 .El 217 .Pp 218 Example: 219 .Bd -literal -offset 4n 220 mkdir -p "$HOME/.sfeed/etags" "$HOME/.sfeed/lastmod" 221 222 # fetch(name, url, feedfile) 223 fetch() { 224 basename="$(basename "$3")" 225 etag="$HOME/.sfeed/etags/${basename}" 226 lastmod="$HOME/.sfeed/lastmod/${basename}" 227 output="${sfeedtmpdir}/feeds/${filename}.xml" 228 229 curl \e 230 -f -s -m 15 \e 231 -L --max-redirs 0 \e 232 -H "User-Agent: sfeed" \e 233 --compressed \e 234 --etag-save "${etag}" --etag-compare "${etag}" \e 235 -R -o "${output}" \e 236 -z "${lastmod}" \e 237 "$2" 2>/dev/null || return 1 238 239 # succesful, but no file written: assume it is OK and Not Modified. 240 [ -e "${output}" ] || return 0 241 242 # use server timestamp from curl -R to set Last-Modified. 243 touch -r "${output}" "${lastmod}" 2>/dev/null 244 cat "${output}" 2>/dev/null 245 # use write output status, other errors are ignored here. 246 fetchstatus="$?" 247 rm -f "${output}" 2>/dev/null 248 return "${fetchstatus}" 249 } 250 .Ed 251 .Pp 252 The README file has more examples. 253 .Sh SEE ALSO 254 .Xr curl 1 , 255 .Xr iconv 1 , 256 .Xr sfeed_update 1 , 257 .Xr sh 1 258 .Sh AUTHORS 259 .An Hiltjo Posthuma Aq Mt hiltjo@codemadness.org