NEWZNAB Usenet Searching Web API v0.3 22/8/2010 Owner: bb at #newznab (irc.synirc.org) info@newznab.com Author: ensi at #newznab (irc.synirc.org) ensisoft@gmail.com 1. Introduction 2. Functions 2.1 CAPS 2.2 REGISTER 2.3 SEARCH 2.4 TV-SEARCH 2.5 MOVIE-SEARCH 2.6 DETAILS 3. Predefined Categories 4. Predefined Attributes 4.1 List of Attributes 4.2 Attribute example 5. Newznab Error Codes 5.1 Error code example 1. Introduction This document describes the NEWZNAB Usenet Searching Web API. The API is designed to be implemented by Usenet indexing sites, i.e. sites that index Usenet newsgroups through some means, typically by downloading and inspecting the NTTP headers. The API is aimed for NZB aware client applications to allow them to perform Usenet searches against Newznab servers and receive NZB information in order to facilitate direct downloading from Usenet without having to download any NTTP headers. This document does not describe the actual implementation of either the client or the server but just describes the HTTP(S) interface and request/response sequences. Intended readers are server and client implementers. 1.1 Notation This document uses the follocation notations: Parameters: "t=c" denotes a required HTTP query parameter. [o=json | o=xml] denotes optional parameters with possible values. 2. Functions All functions are executed as HTTP(S) requests over TCP. All parameters are to be passed as query parmeters unless otherwise indicated. All returned XML/JSON data is UTF-8 encoded unless otherwise specified. All query parameters should be UTF-8 and URL encoded, i.e. query-param = URL-ENCODE(UTF8-ENCODE(param=value)). The functions are divided into two categories. Functions specific to searching and retrieving of items and the their information such as SEARCH and TV-SEARCH and functions that are for site/user account management such as CAPS and REGISTER. Any conforming implementation should support the CAPS and SEARCH functions. Other functions are optional and if not supported will return the "203 Function Not Available" when invoked. 2.1 CAPS Description: CAPS function is used to query the server for supported features and the protocol version and other meta data relevant to the implementation. This function doesn't require the client to provide any login information but can be executed out of "login session". Important fields of the returned data: server/version The version of the protocol implemented by the server. All implementations should be backwards compatible. limits The limit and defaults to the number of search results returned. retention Server retention (how many days NZB information is stored before being purged). category Defines a searchable category which might have any number of subcategories. category/id Unique category ID, can be either one of the standard category IDs or a site specific ID. category/name Any descriptive name for the category. Can be site/language specific. category/description A description of the contents of the category. category/subcat A subcategory. category/subcat/id/ Unique category ID, can be either one of the standard category IDs or a site specific ID. category/subcat/name Any descriptive name for the category. Can be site/language specific. category/subcat/description A description of the contents of the category. HTTP Method: GET HTTP Response: 200 OK Parameters: t=caps Caps function, must always be "caps". Optional parameters: o=xxx Output format, either "JSON" or "XML. Default is "XML". Examples: --> GET http://servername.com/api?t=caps <-- 200 OK 2.2 REGISTER Description: REGISTER function is used for automatically creating and registering user account. This is an optional function and may or may not be available at a site. It is also possible that function is available but currently registrations at the site are closed. The only prerequisite for registering an account is a valid email address and any server policies. It is at the server administration discretion to allow or deny registrations based on for example the validity of the email address or the the current client host address. On succesful registration a valid username, password and api key are returned to the caller On error an appropriate error code is returned. HTTP Method: GET HTTP Response: 200 OK Parameters: t=register Register function, must always be "register" email=xxx A valid email address to be used for registration. (URL/UTF-8 encoded). Examples: --> GET HTTP://servername.com/api?t=register&email=john.joe%40acme.com <-- 200 OK --> GET HTTP://servername.com/api?t=register&email=john.joe%40acme.com <-- 200 OK --> GET HTTP://servername.com/api?t=register&email=john.joe%40acme.com <-- 200 OK --> GET HTTP://servername.com/api?t=register&email=john.joe%40acme.com <-- 200 OK 2.3 SEARCH Description: SEARCH function searches the index for items matching the search criteria. On succesful search the response contains a list of found items. Even if search matched nothing an empty response set is created and returned. This function requires passing the user credentials. Searches that include categories that are not supported by the server are still executed but the non-supported categories are simply skipped. This basically treats such a search simply as a "no match" but allows the same query to be ran simultaneously against several servers. The list of search categories specifies a logical OR condition. I.e. an item matching the search input in any of the specified categories is considered a match and is returned. E.g. a search searching for "linux" in "computer" and "ebook" categories searches for matching items in "computer" and "ebook" but does not search for example the "movies" category. Items found in either group are then combined into a single result set. If the input string for search is empty all items (within the server/query limits) are returned for the matching categories. When performing the query the categories to be searched are concatenated into a single query parameter by , (comma). For example "cat=200,300,400", which is then URL encoded. The returned XML data stream is RSS 2.0 compliant and also contains additional information in the extra namespace. Response-offset field identifies the current subset of all the matches that are being transmitted in the response. In other words, if a search for "disco" finds more matches than the server is capable of transmitting in a single response, the response needs to be split into several responses. Then it is's the clients responsibility to repeat the same query with same parameters but specify an increased offset in order to return the next set of results. If offset query parameter is not used response data contains items between 0 offset - limit. If offset query parameter is out of bounds an empty result set is returned. Important fields of the returned data (RSS): title Title of the found item. guid A globally unique (GUID) item identifier. pubdate The publishing date in RSS date object as specified by RFC822/2822. (not the Usenet date) category The category the NZB belongs to. (This is human readable for RSS. More precise category is found in additional data) enclosure The NZB url HTTP Method: GET HTTP Response: 200 OK Parameters: t=search Search function, must always be "search" apikey=xxxx User's key as provided by the service provider. Optional parameters: q=xxxx Search input (URL/UTF-8 encoded). Case insensitive. limit=123 Upper limit for the number of items to be returned. cat=xxx List of categories to search delimeted by "," o=xxx Output format, either "JSON" or "XML". Default is "XML". extended=1 Return extended information in the search results. (See DETAILS). del=1 Delete the item from a users cart on download. maxage=123 Only return results which were posted to usenet in the last x days. offset=50 The 0 based query offset defining which part of the response we want. Examples: --> GET http://servername.com/api?t=search&apikey=xxxxx&q=a%20tv%20show <-- 200 OK example.com</tile> <description>example.com API results</description> <!-- More RSS content --> <!-- offset is the current offset of the response total is the total number of items found by the query --> <newznab:response offset="0" total="2344"/> <item> <!-- Standard RSS 2.0 Data --> <title>A.Tv.Show.S06E05.DVDRIP.XviD http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c http://servername.com/rss/nzb/e9c515e02346086e3a477a5436d7bc8c&i=1&r=18cf9f0a736041465e3bd521d00a90b9 http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c#comments Sun, 06 Jun 2010 17:29:23 +0100 TV > XviD Some TV show // no items matched the search criteria --> GET http://servername.com/api?t=search&apikey=xxxxx&q=linux%20image <-- 200 OK // Query could not be completed because user credentials are broken --> GET http://servername.com/api?t=search&apikey=xxxxx&q=linux%20image <-- 200 OK // Query could not be completed because it was malformed --> GET http://servername.com/api?t=search&apikey=xxxxx&q=linux%20image <-- 200 OK 2.4 TV-SEARCH Description: TV-SEARCH function searches the index in the TV category for items matching the search criteria. The criteria includes query string and in addition information about season and episode. On succesful search the response contains a list of items that matched the query. Even if the search matched nothing an empty but valid response is created and returned. This function requires passing the user credentials. The returned XML data stream is RSS 2.0 compliant and also contains additional information in the extra namespace and optionally TV specific information HTTP Method: GET HTTP Response: 200 OK Parameters: t=tvsearch TV-Search function, must always be "tvsearch". apikey=xxx User's key as provided by the service provider. Optional parameters: limit=123 Upper limit for the number of items to be returned, e.g. 123. rid=xxxx TVRage id of the item being queried. cat=xxx List of categories to search delimeted by "," season=xxxx Season string, e.g S13 or 13 for the item being queried. q=xxxx Search input (URL/UTF-8 encoded). Case insensitive. ep=xxx Episode string, e.g E13 or 13 for the item being queried. o=xml Output format, either "JSON" or "XML". Default is "XML". extended=1 Return extended information in the search results del=1 Delete the item from a users cart on download. maxage=123 Only return results which were posted to usenet in the last x days. offset=50 The 0 based query offset defining whcih part of the response we want. Examples: --> GET http://servername.com/api?t=tvsearch&apikey=xxxq=lost&season=S03 <-- 200 OK example.com example.com API results Land.of.the.Lost.S03E02.Survival.Kit.iNTERNAL.DVDRip.XViD-SPRiNTER http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c http://servername.com/rss/nzb/e9c515e02346086e3a477a5436d7bc8c&i=1&r=18cf9f0a736041465e3bd521d00a90b9 http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c#comments Sun, 06 Jun 2010 17:29:23 +0100 TV > XviD Some TV show Lost.S03E01.720p.BluRay.DTS.x264.INTERNAL-hV http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c http://servername.com/rss/nzb/e9c515e02346086e3a477a5436d7bc8c&i=1&r=18cf9f0a736041465e3bd521d00a90b9 http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c#comments Sun, 06 Jun 2010 17:29:23 +0100 TV > XviD Some TV show 2.5 MOVIE-SEARCH Description: MOVIE-SEARCH function searches the index for items matching an imdb id or search query. On succesful search the response contains a list of items that matched the query. Even if the search matched nothing an empty but valid response is created and returned. This function requires passing the user credentials. The returned XML data stream is RSS 2.0 compliant and also contains additional information in the extra namespace and optionally movie specific information. HTTP Method: GET HTTP Response: 200 OK Parameters: t=movie Movie-Search function, must always be "movie". apikey=xxx User's key as provided by the service provider. Optional parameters: limit=123 Upper limit for the number of items to be returned, e.g. 123. imdbid=xxxx IMDB id of the item being queried e.g. 0058935. cat=xxx List of categories to search delimeted by "," q=xxxx Search input (URL/UTF-8 encoded). Case insensitive. o=xml Output format, either "JSON" or "XML". Default is "XML". extended=1 Return extended information in the search results del=1 Delete the item from a users cart on download. maxage=123 Only return results which were posted to usenet in the last x days. offset=50 The 0 based query offset defining whcih part of the response we want. Examples: --> GET http://servername.com/api?t=movie&apikey=xxx&imdbid=0058935 <-- 200 OK example.com example.com API results Movie.Name.720p.BluRay.DTS.x264 http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c http://servername.com/rss/nzb/e9c515e02346086e3a477a5436d7bc8c&i=1&r=18cf9f0a736041465e3bd521d00a90b9 http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c#comments Sun, 06 Jun 2010 17:29:23 +0100 Movie > XviD Some movie 2.6 DETAILS Description: DETAILS function returns all information for a particular Usenet (NZB) item. The response contains the generic RSS part + full extra information + full type/category specific information. HTTP Method: GET HTTP Response: 200 OK Parameters: t=details Details function, must always be "details". guid=xxxx The GUID of the item being queried. apikey=xxxx User's key as provided by the service provider. Optional parameters: o=xxx Output format, either "JSON" or "XML". Default is "XML". del=1 Delete the item from a users cart on download. Example: --> GET http://servername.com/api?t=details&apikey=xxxxx&guid=xxxxxxxxx <-- 200 OK A.Tv.Show.S06E05.DVDRIP.XviD http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c http://servername.com/rss/nzb/e9c515e02346086e3a477a5436d7bc8c&i=1&r=18cf9f0a736041465e3bd521d00a90b9 http://servername.com/rss/viewnzb/e9c515e02346086e3a477a5436d7bc8c#comments Sun, 06 Jun 2010 17:29:23 +0100 TV > XviD Some TV show // Query could not be completed because it was malformed --> GET http://servername.com/api?t=details&apikey=xxxxx&guid=xxxxxxxxx <-- 200 OK // Query could not be completed because no such item was available --> GET http://servername.com/api?t=details&apikey=xxxxx&guid=xxxxxxxxx <-- 200 OK // Query could not be completed because user credentials are broken --> GET http://servername.com/api?t=details&apikey=xxxxx&guid=xxxxxxxxx <-- 200 OK 3. Predefined Categories In order to facilitate operation that does not rely on a particular natural language, e.g. english a set of predefined catetegory IDs have been defined. It is possible to define custom categories in the custom category range. Each category is given a range for a set of subcategories. It is possible for an item to belong to several categories at the same time. Category Range Category Name Comments 0000-0999 Reserved 1000-1999 Console 2000-2999 Movies 3000-3999 Audio 4000-4999 PC 5000-5999 TV 6000-6999 XXX 7000-7999 Other 8000-99999 Reserved Reserved for future expansion 100000- Custom Site specific category range. Defined in CAPS Categories Category Name 0000 Reserved 1000 Console All of console 1010 Console/NDS Nintendo DS 1020 Console/PSP Sony Playstation Portable 1030 Console/Wii Nintendo Wii 1040 Console/XBox Microsoft XBox 1050 Console/Xb0x 360 Microsoft XBox 360 1060 Console/Wiiware Wii homebrew 1070 Console/Xb0x 360 DLC Microsoft XBox 360 Downloadable Content 2000 Movies All of movies 2010 Movies/Foreign 2020 Movies/Other 2030 Movies/SD 2040 Movies/HD HD movies with AVC encoding 3000 Audio All of audio 3010 Audio/MP3 3020 Audio/Video Music videos 3030 Audio/Audiobook wtf is an audiobook? 3040 Audio/Lossless 4000 PC All of PC 4010 PC/0day Apps and games not released in ISO. 4020 PC/ISO CD-ROM images/DVD Images 4030 PC/Mac 4040 PC/Phone Mobile phone software 4050 PC/Games PC Games 5000 TV All of TV 5020 TV/FOREIGN FOREIGN TV 5030 TV/SD SD TV 5040 TV/HD HD TV 5050 TV/OTHER Other TV Content 5060 TV/Sport 6000 XXX All of XXX 6010 XXX/DVD 6020 XXX/WMV 6030 XXX/XviD 6040 XXX/x264 HD Pron 7000 Other All of other 7010 Misc 7020 EBook E-Books 7030 Comics Comics Ebooks 100000- Custom Specific to a site 4. PREDEFINED ATTRIBUTES A set of known attributes for items in different categories has been defined. Its possible that not all attributes are vailable at all times. Therefore a client application should not rely on any particular attributes being in the returned data but should take this list as an optional extra information. However attributes marked with * are always available. Additionally, not all attributes are applicable to all items. The category information can be used to check which attributes area available for which category items. All attributes are defined using XML namespace syntax. e.g. xmlns:newznab="http://www.newznab.com/DTD/2010/feeds/attributes/" 4.1 List of Attributes Attribute Category Description Example value size * ALL Size in bytes "252322" category * ALL Item's category "5004" files ALL Number of files "4" poster ALL NNTP Poster "yenc@power-post" group ALL NNTP Group(s) "a.b.warez, a.b.teevee" team ALL Team doing the release "DiAMOND" grabs ALL Number of times item downloaded "1" password ALL Whether the archive is passworded "0" no, "1" rar pass, "2" contains inner archive comments ALL Number of comments "2" usenetdate ALL Date posted to usenet "Tue, 22 Jun 2010 06:54:22 +0100" info ALL Info (.nfo) file URL "http://somesite/stuff/info.php?id=1234" year ALL Release year "2009" season TV Numeric season "1" episode TV Numeric episode within the season "1" rageid TV TVRage ID. (www.tvrage.com) "2322" video TV, Movies Video codec "x264" audio TV, Movies, Audio Audio codec "AC3 2.0 @ 384 kbs" resolution TV, Movies Video resolution "1280x716 1.78:1" framerate TV, Movies Video fps "23.976 fps" language TV, Movies, Audio Natural languages "English" subs TV, Movies Subtitles "English, Spanish" imdb TV, Movies IMDb ID (www.imdb.com) "0104409" genre TV, Movies Genre "Horror" 4.2 Attribute Example Example attribute declarations within element. 5. Newznab Error Codes Under normal circumstances i.e. when the HTTP requst/response sequence is succesfully completed Newznab implementations always respond with HTTP 200 OK. However this doesn't mean that the query was semantically correct. It simply means that the HTTP part of the sequence was succesful. One then must check the actual response body/data to see if the request was completed without errors. In case of a Newznab error the response contains an error code and an a description of the error. The error codes have been defined into different ranges. 100-199 Account/user credentials specific error codes, 200-299 API call specific error codes, 300-399 content specific error codes and finally 900-999 Other error codes. Error code Description 100 Incorrect user credentials 101 Account suspended 102 Insufficient priviledges/not authorized 103 Registration denied 104 Registrations are closed 105 Invalid registration (Email Address Taken) 106 Invalid registration (Email Address Bad Format) 107 Registration Failed (Data error) 200 Missing parameter 201 Incorrect parameter 202 No such function. (Function not defined in this specification). 203 Function not available. (Optional function is not implemented). 300 No such item. 900 Unknown error 5.1 Error code example // Query could not be completed because user credentials are broken --> GET http://servername.com/api?t=details&apikey=xxxxx&guid=xxxxxxxxx <-- 200 OK