A specifies one or more search patterns, being given as a single character, a character vector, a vector of character vectors or a vector of both characters and character vectors. See search pattern following.

B is the transformation to be performed on matches within the input document; it may be either one or more transformation patterns (specified as a character, a character vector, a vector of character vectors, or a vector of both characters and character vectors), one or more transformation codes (specified as a numeric scalar or a numeric vector) or a function; see transformation pattern, transformation codes and transformation function following.

Y specifies the input document; see input document below.

X optionally specifies an output stream; see output below.

R is the result value; see output below.

      ('.at' ⎕R '\u0') 'The cat sat on the mat'     
The CAT SAT on the MAT

      ('\w+' ⎕R {⌽⍵.Match}) 'The cat sat on the mat'
ehT tac tas no eht tam

      STR←'The cat sat on the mat'
      ('.at' ⎕S '\u0') STR
CAT  SAT  MAT 

      ('.at' ⎕S {⍵.((1↑Offsets),1↑Lengths)}) STR
4 3  8 3  19 3

      ('.at' ⎕S 0 1) STR       
4 3  8 3  19 3

These examples all operate on a simple character vector containing text, but the text may be given in several forms - character vectors, vectors of character vectors, and external data streams. These various forms constitute a 'document'. When the result also takes the form of a document it may be directed to a stream.

Currently, the only supported data stream is a native file, specified as tie number, which is read from the current position to the end. If the file is read from the start, and there is a valid Byte Order Mark (BOM) at the start of it, the data encoding is determined by this BOM. Otherwise, data in the file is assumed to be encoded as specified by the InEnc option.

      {} ⎕NREAD tienum 82 0 0

The implicit line ending character may be set using the EOL option. Explicit line ending characters may also be replaced by this character - so that all line endings are normalised - using the NEOL option.

The input document may be processed in line mode, document mode or mixed mode. In document mode and mixed mode, the entire input document, line ending characters included, is passed to the search engine; in line mode the document is split on line endings and passed to the search engine in sections without the line ending characters. The choice of mode affects both memory usage and behaviour, as documented in the section 'Line, document and mixed modes'.

An output data stream may optionally be specified. Currently, the only supported data stream is a native file, specified as tie number, and all output will be appended to it. Data in the stream is encoded as specified by the OutEnc option. If this encoding specifies a Byte Order Mark and the file is initially empty then the Byte Order Mark will be written at the start. Appending to existing data using a different encoding is permitted but unlikely to produce desirable results. If an input stream is also used, care must be taken to ensure the input and output streams are not the same.

⎕R

With no output stream specified and unless overridden by the ResultText option, the derived function result will be a document which closely matches the format of the input document, as follows:

A character scalar or vector input will result in a character vector output. Any and all line endings in the output will be represented by line ending characters within the character vector.

A vector of character vectors as input will result in a vector of character vectors as document output. Any and all line endings in the output document will be implied at the end of each character vector. 

A stream as input will result in a vector of character vectors document output. Any and all line endings in the output document will be implied at the end of each character vector. 

Note that the shape of the output document may be significantly different to that of the input document.

If the ResultText option is specified, the output type may be forced to be a character vector or vector of character vectors as described above, regardless of the input document.

With an output stream specified the text is appended to the stream. If the appended text does not end with a line ending character then the line ending character specified by the EOL option is also appended. The resulting length of the file is returned as a shy result.

⎕S

With no output stream specified, the result will be a vector containing one item for each match in the input document, of types determined by the transformation performed on each match.

With an output stream specified each match is appended to the stream. If any match does not end with a line ending character then the line ending character specified by the EOL option is also appended. The resulting length of the file is returned as a shy result. Only text may be written to the stream, which means:

There may be multiple search patterns. If more than one search pattern is specified and more than one pattern matches the same part of the input document then priority is given to the pattern specified first.

Note that when anchoring a search to the beginning of a line, it is essential to use ^ (⎕UCS 94), not ∧ (⎕UCS 8743).

Transformation pattern

For each match in the input document, the transformation pattern causes the creation of text which, for ⎕R, replaces the matching text and, for ⎕S, generates one item in the result.

There may be either one transformation pattern, or the same number of transformation patterns as search patterns. If there are multiple search patterns and multiple transformation patterns then the transformation pattern used corresponds to the search pattern which matched the input text.

Transformation patterns may not be mixed with transformation codes or functions.

The following characters have special meaning:

Character sequences beginning with the backslash other that those shown are invalid. All characters other than those shown are literal values and are included in the text without modification.

Transformation codes

The return value from the function is used as follows:

With ⎕R the function must return a character vector. The contents of this vector are used to replace the matching text.

The passed namespace exists over the lifetime of ⎕R or ⎕S; the function may therefore preserve state by creating variables in the namespace.

The function may itself call ⎕R or ⎕S.

There may be only one transformation function, regardless of the number of search patterns.

The locations of the match within Block and subpatterns within Match are given as offsets rather than positions, that is, the values are the number of characters preceding the data, and are not affected by the Index Origin.

The value of ¯1 may appear in both the Offsets and Fields items (in corresponding positions). They indicate that the subpattern to which they refer did not appear in the match at all.

Example

      {}('(A)|(B)'⎕R{⎕←⍵.(Offsets Lengths)⋄'x'})'ABC'
 0 0  1 1 
 1 ¯1 1  1 ¯1 1 

The pattern has two subpatterns - (A) and (B). Therefore Offsets and Lengths would be expected to have three elements each - one for the entire match, one for the first subpattern and one for the second subpattern. But these subpatterns have a | between them which means they are alternates - only one can match.

When ABC is searched the first match is the A in the first subpattern. The second subpattern does not feature. Offsets is 0 0 and Lengths is 1 1: the entire pattern matched from offset 0 length 1 and the first subpattern also matched from offset 0 length 1. The second subpattern did not feature in the match.

Note that ¯1 is only used as a "filler" when there are higher-numbered subpatterns that did match.

The second match is the B in the second subpattern. Offsets is 1 ¯1 1 and Lengths is 1 ¯1 1: the entire pattern matched from offset 1 length 1 and the second subpattern also matched from offset 1 length 1. The first subpattern did not feature in the match and this is indicated by the ¯1s. There has to be something between the offset/length for the entire pattern and the second subpattern.

      ('[AEIOU]' ⎕R 'X' ⍠ 'IC' 1) 'ABCDE abcde'
XBCDX XbcdX
      ('[AEIOU]' ⎕R 'X' ⍠ 1)'ABCDE abcde'
XBCDX XbcdX

Examples:

      ('$' ⎕R '[Endline]' ⍠ 'Mode' 'L') 'ABC' 'DEF'
 ABC[Endline]  DEF[Endline] 
        
      ('$' ⎕R '[Endline]' ⍠ 'Mode' 'D') 'ABC' 'DEF'
 ABC  DEF[Endline]
         
      ('$' ⎕R '[Endline]' ⍠ 'Mode' 'M') 'ABC' 'DEF'
 ABC[Endline]  DEF[Endline]

      ('.' ⎕R 'X' ⍠'Mode' 'D') 'ABC' 'DEF'
 XXX  XXX 
      ('.' ⎕R 'X' ⍠('Mode' 'D')('DotAll' 1)) 'ABC' 'DEF'
 XXXXXXXX

EOL Option

Sets the line ending character which is implicitly present between character vectors, when the input document is a vector of character vectors.

In the Classic Edition, setting a value which is not in ⎕AVU may result in a TRANSLATION ERROR.

Example:

      ('\n' ⎕R'X' ⍠('Mode' 'D')('EOL' 'LF')) 'ABC' 'DEF'
 ABCXDEF

Here, the implied line ending between 'ABC' and 'DEF' is '\n', not the default '\r\n'.

      a←'ABC',(1↑2↓⎕AV),'DEF',(1↑3↓⎕AV),'GHI'
      ('\n'⎕S 0 ⍠ 'Mode' 'D' ⍠ 'NEOL' 1 ⍠ 'EOL' 'LF') a
3 7

      ('.' ⎕R 'x' ⍠ 'ML' 2) 'ABC' 'DEF'
 xxC  xxF 
      ('.' ⎕R 'x' ⍠ 'ML' ¯2) 'ABC' 'DEF'
 AxC  DxF 
      ('.' ⎕R 'x' ⍠ 'ML' ¯4 ⍠ 'Mode' 'D') 'ABC' 'DEF'
 ABC  xEF

      ('[A-Z].*[0-9]' ⎕R 'X' ⍠ 'Greedy' 1)'ABC123 DEF456'
X
      ('[A-Z].*[0-9]' ⎕R 'X' ⍠ 'Greedy' 0)'ABC123 DEF456'
X23 X56

      ('[0-9]+' ⎕S '\0' ⍠ 'OM' 0) 'A 1234 5678 B'
 1234  5678 
      ('[0-9]+' ⎕S '\0' ⍠ 'OM' 1) 'A 1234 5678 B'
 1234  234  34  4  5678  678  78  8

      ⎕UCS ¨ ('A' ⎕R 'x') 'AB' 'CD'                
  120 66  67 68                    
      ⎕UCS ('A' ⎕R 'x' ⍠ 'ResultText' 'Simple') 'AB' 'CD'
 120 66 13 10 67 68

      ('\w'⎕S'\0')'Bjørn'  ⍝ identify "word" characters
 B  j  r  n
      ('\W'⎕S'\0')'Bjørn'  ⍝ non-word" characters
 ø

      ('\w'⎕S'\0' ⍠'UCP' 1)'Bjørn'
 B  j  ø  r  n

Examples

      STR
The cat sat on the mat

      ('.at' ⎕S '\u0') STR
 CAT  SAT  MAT 
      ('.at' ⎕S '\u0' ⍠('Regex' 0)) STR

      ('.at'⎕R'\u0')STR
The CAT SAT on the MAT
      ('.at'⎕R'\u0'⍠('Regex' (1 0)))STR
The \u0 \u0 on the \u0

Memory usage differences

Blocks of data passed to the search engine are processed and stored in the workspace. Processing the input document in line mode limits the total memory requirements; in particular this means that large streams can be processed without holding all the data in the workspace at the same time.

Technical Considerations

⎕R and ⎕S utilise the open-source regular-expression search engine PCRE, which is built into the Dyalog software and distributed according to the PCRE licence which is published separately.

Before data is passed to PCRE it is converted to UTF-8 format. This converted data is buffered in the workspace; processing large documents may have significant memory requirements. In line mode, the data is broken into individual lines and each is processed separately, potentially reducing memory demands.

It is possible to save a workspace with an active ⎕R or ⎕S on the stack and execution can continue when the workspace is reloaded with the same interpreter version. Later versions of the interpreter may not remain compatible and may signal a DOMAIN ERROR with explanatory message in the status window if it is unable to continue execution.

PCRE has a buffer length limit of 2³¹ bytes (2GB). UTF-8 encodes each character using between 1 and 6 bytes (typically 1 or 3). In the very worst case, where every character is encoded in 6 bytes, the maximum block length which can be searched would be 357,913,940 characters.

Further Examples

      text
To be or not to be- that is the question:
Whether 'tis nobler in the mind to suffer
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles

Replace all upper and lower-case vowels by 'X':

       ('[aeiou]' ⎕R 'X' ⍠ 'IC' 1) text
TX bX Xr nXt tX bX- thXt Xs thX qXXstXXn:   
WhXthXr 'tXs nXblXr Xn thX mXnd tX sXffXr   
ThX slXngs Xnd XrrXws Xf XXtrXgXXXs fXrtXnX,
Xr tX tXkX Xrms XgXXnst X sXX Xf trXXblXs   

Replace only the second vowel on each line by '\VOWEL\':

       ('[aeiou]' ⎕R '\\VOWEL\\'⍠('IC' 1)('ML' ¯2)) text
To b\VOWEL\ or not to be- that is the question:   
Wheth\VOWEL\r 'tis nobler in the mind to suffer   
The sl\VOWEL\ngs and arrows of outrageous fortune,
Or t\VOWEL\ take arms against a sea of troubles

Case fold each word:

      ('(?<first>\w)(?<remainder>\w*)' ⎕R '\u<first>\l<remainder>') text
To Be Or Not To Be- That Is The Question:   
Whether 'Tis Nobler In The Mind To Suffer   
The Slings And Arrows Of Outrageous Fortune,
Or To Take Arms Against A Sea Of Troubles   

Extract only the lines with characters 'or' (in upper or lower case) on them:

      ↑('or' ⎕S '%' ⍠ ('IC' 1)('ML' 1)) text
To be or not to be– that is the question:   
The slings and arrows of outrageous fortune,
Or to take arms against a sea of troubles 

Identify which lines contain the word 'or' (in upper or lower case) on them:

      ('\bor\b'⎕S 2⍠('IC' 1)('ML' 1))text
0 3

Note the difference between the characters 'or' (which appear in 'fortune') and the word 'or'.

Place every non-space sequence of characters in brackets:

      ('[^\s]+' ⎕R '(&)' ) 'To be or not to be, that is the question'
(To) (be) (or) (not) (to) (be,) (that) (is) (the) (question)

Replace all sequences of one or more spaces by newline. Note that the effect of this is dependent on the input format:

Character vector input results in a single character vector output with embedded newlines:

      ]display ('\s+' ⎕R '\r') 'To be or not to be, that is the question'
┌→───────┐
│To      │
│be      │
│or      │
│not     │
│to      │
│be,     │
│that    │
│is      │
│the     │
│question│
└────────┘

      ]display ('\s+' ⎕R '\r') 'To be or not to be,' 'that is the question'
┌→─────────────────────────────────────────────────────────────┐
│ ┌→─┐ ┌→─┐ ┌→─┐ ┌→──┐ ┌→─┐ ┌→──┐ ┌→───┐ ┌→─┐ ┌→──┐ ┌→───────┐ │
│ │To│ │be│ │or│ │not│ │to│ │be,│ │that│ │is│ │the│ │question│ │
│ └──┘ └──┘ └──┘ └───┘ └──┘ └───┘ └────┘ └──┘ └───┘ └────────┘ │
└∊─────────────────────────────────────────────────────────────┘

Change numerals to their expanded names, using a function:

     ∇r←f a;n
[1]   n←'zero' 'one' 'two' 'three' 'four'
[2]   n,←'five' 'six' 'seven' 'eight' 'nine'
[3]   r←' ',⊃(⍎a.Match)↓n
     ∇
      verbose←('[0-9]' ⎕R f)
      verbose ⍕27×56×87
 one three one five four four

Swap 'red' and 'blue':

      ('red' 'blue' ⎕R 'blue' 'red') 'red hat blue coat'
blue hat red coat

Convert a comma separated values (CSV) file so that

• dates in the first field are converted from European format to ISO, and
• currency values are converted from Deutsche Marks (DEM) to Euros (DEM 1.95583 to €1).

The currency conversion requires the use of a function. Note the nested use of ⎕R.

Input file:

Output file:

     ∇ ret←f a;d;m;y;v
[1]    ⎕IO←0
[2]    :Select a.PatternNum
[3]    :Case 0
[4]        d m y←{a.Match[a.Offsets[⍵+1]+⍳a.Lengths[⍵+1]]}¨⍳3
[5]        ret←y,'-',m,'-',d,','
[6]    :Else
[7]        v←⍎a.Block[a.Offsets[1]+⍳a.Lengths[1]]
[8]        v÷←1.95583
[9]        ret←',€ ',('(\d+\.\d\d).*'⎕R'\1')⍕v
[10]   :EndSelect
     ∇

      in ← 'x.csv' ⎕NTIE 0
      out ← 'new.csv' ⎕NCREATE 0
      dateptn←'(\d{2})/(\d{2})/(\d{4}),'
      valptn←',DEM ([0-9.]+)'
      out (dateptn valptn ⎕R f) in
      ⎕nuntie¨in out

Create a simple profanity filter. For the list of objectionable words:

       profanity←'bleeding' 'heck'

first construct a pattern which will match the words:

      ptn←(('^' '$' '\r\n') ⎕R '\\b(' ')\\b' '|'
                           ⎕OPT 'Mode' 'D') profanity
      ptn
\b(bleeding|heck)\b

then a function that uses this pattern:

      sanitise←ptn ⎕R '****' ⎕opt 1
      sanitise '"Heck", I said'
"****", I said

Replace the characters 'or' with '\u0' without having to escape the backslash:

Escaping transformation strings can be a daunting task. To avoid doing so, one can simply enclose the string in braces. This is not a special feature, but just a consequence of how transformation functions are used.

      ('to' ⎕R {'\u0'})text
To be or not \u0 be– that is the question:
Whether 'tis nobler in the mind \u0 suffer
The slings and arrows of outrageous fortune,
Or \u0 take arms against a sea of troubles