LinkOut XML Files from Providers

LinkOut information is submitted by link providers in XML. LinkOut XML is defined by the LinkOut Document Type Definition (DTD).

The LinkOut DTD specifies all the elements needed to build the logical data unit for the LinkOut database. The DTD also specifies the link provider information needed for future communication with the providers regarding their links.

Two root elements are specified in the LinkOut DTD: the <Provider> element, which specifies information about a link provider; and the <LinkSet> element, which describes information about the link. Each root element is submitted to NCBI in a separate file. An identity file contains the <Provider> element, and a resource file contains the <LinkSet> element.

The identity file, providerinfo.xml, describes the identity of a provider, including an ID <ProviderId>, an abbreviated name <NameAbbr> assigned by NCBI, the provider's name, and other general information about the provider. There is only one providerinfo.xml file for each provider (see the details of the identity file).

The resource file, which contains the linking information, specifies a set of Entrez database records by UIDs or a valid query to the database, a specific rule to build the URL to an external resource, and the description of the resource using the SubjectType, Attribute, and UrlName fields. There is no standard for naming resource files, except that they must use the .xml extension. There may be any number of resources files associated with a ProviderId (see the details of the resource file).

Preprocessing

LinkOut files from providers are passed through a preprocessing stage that converts them to standardized files for the LinkOut indexer.

The preprocessing stage includes three sub-processes:

1.

Loading—Files submitted by providers in their FTP accounts are copied over to a staging area for the LinkOut indexer. During this sub-process, the XML files are adjusted to make sure the information in the file is valid. For example, ProviderId is a valid ID assigned by NCBI and only terms from the control lists of SubjectType and Attribute are used. The adjusted files are also validated against the LinkOut DTD to ensure XML files passed to the LinkOut indexer are with valid syntax. The loading sub-process compares the XML files in a provider’s FTP account and the files in the staging area and decides which files in the FTP account should be processed. Only new and changed files in a provider’s FTP account will be processed.

2.

XML files generation—Holdings information entered into the Library Submission Utility are transformed into LinkOut XML files and added to the loading sub-process discussed above. Only holdings from libraries that have changed holdings information are processed.

3.

Icon processing—Icons specified in the <IconUrl> element are downloaded. The icons downloaded are adjusted to make sure their size is within the allowed dimensions specified by NCBI. The processed icons are transferred to the target databases to form a part of the record display.

LinkOut Indexing

The LinkOut indexing process takes the standardized LinkOut XML files in the staging area and converts them into the standard XML suitable for Entrez indexing.

This process converts all queries specified in LinkOut file <Query> elements to UIDs of the target database. Since a <Query> in a LinkOut file can be translated to a different set of UIDs as the target database changes over time, it is necessary to process all LinkOut files in each LinkOut indexing. For example, <Query>plos one [journal]<Query> will be translated to a different set of PubMed IDs as citations for PLOS One are added to PubMed between LinkOut indexing.

LinkOut indexing is optimized to execute duplicate queries across all LinkOut files only once. In doing so, the number of queries needed to be processed has been reduced significantly and the speed of LinkOut indexing has been improved by tenfold.

Entrez Indexing

The Entrez indexing process builds the LinkOut database using the LinkOut files in the standard Entrez indexing format. The unit of the LinkOut database corresponds to the logical data unit described in the Data Model section. The LinkOut database can be queried directly with Entrez search commands internally at NCBI. It is also the data source for the LinkOut backend program. During the Entrez indexing process, LinkOut filters are generated for each target database. A LinkOut filter is a list of UIDs of a target database with a name. Filters are sent to each target database to be included during the indexing process for the database. As a result, the filters can be searchable in the target database. See the Access section for the details on LinkOut filters.

LinkOut Backend and Frontend of the Target Database

LinkOut backend handles the interaction between the LinkOut database and the frontend program of the NCBI databases. It has standardized protocols and commands that allow access to linking data in the LinkOut database more effectively and efficiently.

Each Entrez database may present LinkOut information to users differently. Generally, a frontend program accesses the LinkOut backend to find out:

1.

If a specific record has any LinkOut links;

2.

If there are LinkOut links, the information for each link.

The frontend program uses the information returned by the LinkOut backend and the XSLT rules set by the target database to generate the display. The frontend program is also responsible for transforming the LinkOut entities returned by the LinkOut backend to the value for a specific record to form a valid URL.

For example, in NCBI sequence databases, &lo.pacc; translates into the Primary Access Number of the corresponding record. In the Nucleotide database, record GI: 467096719, the following linking rule:

http://genome.ucsc.edu/cgi-bin/hgTracks?org=human&position=&lo.pacc;

will be translated into the following URL:

http://genome.ucsc.edu/cgi-bin/hgTracks?org=human&position=NR_102347

LinkOut Filters

To facilitate search and retrieval of LinkOut resources, there are a number of filters in the LinkOut-enabled Entrez databases. These filters, although not part of the LinkOut database, are based on the results generated in the LinkOut indexing process. A LinkOut filter is a list of UIDs for a target Entrez database. The UIDs identify records with a common property in the target database. This property is reflected in the filter name.

LinkOut filters are all prefixed with lo. Filters are available for all allowable SubjectType and Attribute terms, and the NameAbbr of a provider. To retrieve a set of records by a certain LinkOut property, a filter name can be entered as a search term in a database search box.

Examples of LinkOut filters include:

• loprov—Filter for records with links to a specific LinkOut provider. Example:
  • “loprovplos” [filter]
  • Retrieves all records with links to the journal Public Library of Science in PubMed.
• loattr—Filter for records with links of a specific LinkOut Attribute. Example:
  • “loattrfull text online” [filter]
  • Retrieves all records with at least one full text link in PubMed.
• losubjt—Filter for records with links of a specific LinkOut SubjectType. Example:
  • “losubjtorganism specific”[filter]
  • Retrieves all records with links to resources of the SubjectType “organism specific”, i.e., resource in a database providing data specific to a particular organism or group of organisms.
• loall—Filter for all records in a database with at least one LinkOut resource. Example:
  • “ loall”[filter]

The Advanced Search Builder of each database can be used to retrieve LinkOut filters. Select Filter, type “lo”, and then click the show index list link to browse through the filters related to LinkOut.

Figure 4:

LinkOut Filters in Advanced Search Builder

Users can also customize LinkOut filters and icons by setting the Filters preferences in My NCBI.

Figure 5:

My NCBI Filters