text
abashin6www.datamystic.com TextPipe Restrictions Page 1
TEXTPIPE RESTRICTIONS -
WHAT ARE THEY AND HOW DO I USE THEM?
What is a restriction?.........................................................................................................................1
Creating Subfilters..........................................................................................................................1
Combining Restrictions......................................................................................................................2
Restrict to line and column ranges...................................................................................................3
Restrict to CSV and Tab-delimited field ranges...............................................................................3
HTML and XML restrictions...............................................................................................................3
Restricting to sections of a .INI file...................................................................................................3
Restrict to lines matching/not matching a pattern...........................................................................4
Restrict to the filename......................................................................................................................4
Restrict to each line in turn................................................................................................................4
Restrict to lines with X Tab fields/CSV fields...................................................................................4
Restrict to blocks of length XXX.......................................................................................................5
Splitting log files.................................................................................................................................5
Feedback and Questions..................................................................................................................7
More White Papers and Documentation Like This..........................................................................7
TextPipe Pro Evaluation....................................................................................................................7
Contact Details...................................................................................................................................7
What is a restriction?
TextPipe Restrictions are one of its most powerful features. Put simply, they allow you to control
exactly where a text modification occurs. An example of a restriction could be lines 1 to 10, or
columns 3 to 17, or only lines matching a pattern. For example, if you need to perform a search
and replace in only the very first line of a file (e.g. the header line of a CSV or tab-delimited file),
you could do it like this:
|--Restrict lines:Line 1 .. line 1
| |
| +--Replace [old value] with [new value]
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [ ] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [ ] Extract matches
2 By the way, if you've never seen filters displayed as above, check out the File Menu\Export
Filter to Clipboard option)
Creating Subfilters
PDF created with pdfFactory trial version www.pdffactory.com
www.datamystic.com TextPipe Restrictions Page 2
A subfilter is where one filter appears inside another, just like subfolders do in Windows Explorer.
In TextPipe, a subfilter indicates a constraint or restriction of some form. You can drop filters
inside a comment filter so that both the documentation and the filters are kept together.
To create a subfilter:
1. First add both filters from the Filter Menu
2. Drag and drop the lower filter to the RIGHT HAND HALF of the upper filter in the filter
view.
3. When you drop it, it will be added as a subfilter, inside the upper filter.
You can check the status line (at the bottom of the screen) to see where TextPipe is going to add
the dropped filter. In general, dropping a filter in the right hand half of the filter view drops it as a
subfilter (if the filter you are dropping it on supports subfilters), and dropping it in the left hand half
of the filter view drops it underneath but at the same level as the filter being dropped on.
You can read more about subfilters in Subfilters - What are they and how do I use them?
www.datamystic.com/docs
Combining Restrictions
You can combine restrictions by putting one restriction inside another. In this example, we only
want to reformat those lines of a report that contain the word 'Reference' and also contain the
word 'Page':
|--Restrict to lines matching [REFERENCE]
| | [ ] Include line numbers
| | [ ] Include filename
| | [ ] Match case
| | [ ] Count matches
| | Pattern type: 0
| | Context before: 0
| | Context after: 0
| |
| +--Restrict to lines matching [PAGE]
| | [ ] Include line numbers
| | [ ] Include filename
| | [ ] Match case
| | [ ] Count matches
| | Pattern type: 0
| | Context before: 0
| | Context after: 0
| |
| |--Remove blanks from Start of Line
| |
| |--Remove multiple whitespace
| |
| +--Replace [REFERENCE] with [Reference]
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [ ] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [ ] Extract matches
PDF created with pdfFactory trial version www.pdffactory.com
www.datamystic.com TextPipe Restrictions Page 3
Restrict to line and column ranges
Another example - you can restrict to Columns 2-20 of lines 1-100 like this:
|--Restrict lines:Line 1 .. line 100
| |
| +--Restrict columns:Column 2 .. column 20
| |
| +--Do something here...
2 Having a Restrict Lines inside a Restrict Columns doesn't work because a column range only
consists of one line at a time.
Restrict to CSV and Tab-delimited field ranges
You can also restrict to CSV or Tab-delimited field ranges, to modify field values inside one or
more fields -
|--Restrict CSV fields:CSV field 1 .. field 1
| |
| +--Convert to lowercase
HTML and XML restrictions
Let's say you wanted to rename an HTML FORM INPUT field from 'name' to 'surname', without
affecting anything else:
|--Restrict to between tags <form>...</form>
| | [ ] Include text
| | [ ] Match case
| |
| +--Restrict to tag <input...>
| | [ ] Include text
| | [ ] Match case
| |
| +--Replace [name] with [surname]
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [ ] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [ ] Extract matches
Although we've omitted it for clarity, you could also add a 'Restrict to attribute' filter to ensure the
replacement only occurred inside the NAME attribute of the INPUT tag.
Restricting to sections of a .INI file
You can restrict a replacement or set of changes to a single section of a .INI file. The trick is to
use a search/replace as the restriction. The example below converts a .INI file section starting
with [TmainForm] to lowercase. This can be easily changed to perform a search/replace, or to
remove matching lines, or to add new lines with an Add Footer filter.
PDF created with pdfFactory trial version www.pdffactory.com
www.datamystic.com TextPipe Restrictions Page 4
EasyPattern [[ mustBeginWith( '[TmainForm]', cr, lf, ), 1+ char,
mustEndWith( cr, lf, '[' ) ]] with [$0]
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [X] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [ ] Extract matches
| Maximum text buffer size 4096
|
+--Convert to lowercase
If the .INI file section will be longer than 4Kbytes, increase the maximum text buffer size to suit.
Restrict to lines matching/not matching a pattern
There are two restriction filters that can be used to limit to lines matching or not matching a perl or
EasyPattern pattern. In the example below, we remove columns 1 to 15 of lines starting with the
word 'HD2'
|--Restrict to lines matching [^HD2]
| | [ ] Include line numbers
| | [ ] Include filename
| | [ ] Match case
| | [ ] Count matches
| | Pattern type: 0
| | Context before: 0
| | Context after: 0
| |
| +--Remove column:Column 1 .. column 15
Restrict to the filename
You can perform search and replace, numbering and other functions inside the filename -
|--Restrict to file name
| |
| +--Convert to Title Case
Restrict to each line in turn
This filter is handy if you want to perform a search and replace against each line without the
possibility of a new line being included in the found text.
Restrict to lines with X Tab fields/CSV fields
Using EasyPatterns, you can easily setup a restriction that will only match lines with a given
number of CSV or Tab fields.
Using a Filters Menu\Restrict\Matching Lines filter, use one of the following patterns:
Description EasyPattern (*)
Match lines with exactly 4 Tab fields [ lineStart, 3 (TabField, Tab), TabField, lineEnd ]
Match lines with 6 or more Tab fields [ lineStart, 5+ (TabField, Tab), TabField, lineEnd ]
Match lines with 3 or less Tab fields [ lineStart, 0 to 2 (TabField, Tab), TabField, lineEnd ]
PDF created with pdfFactory trial version www.pdffactory.com
www.datamystic.com TextPipe Restrictions Page 5
Match lines with exactly 4 CSV fields [ lineStart, 3 (CSVField, Comma), CSVField, lineEnd ]
Match lines with 6 or more CSV fields [ lineStart, 5+ (CSVField, Comma), CSVField,
lineEnd ]
Match lines with 3 or less CSV fields [ lineStart, 0 to 2 (CSVField, Comma), CSVField,
lineEnd ]
* Note how the number is generally set to one less than the number of fields required, because
with n fields there are (n-1) fields each followed by a Tab/Comma, then one more field without a
Tab/Comma.
Restrict to blocks of length XXX
This filter is handy for processing mainframe fixed block length reports, which don't have end of
line characters.
The example below is for processing a mainframe report with blocks of length 764. Starting at
column 164, there are 8 consecutive fields containing packed decimal values that must be
expanded. Each field is 8 characters long.
|--Restrict to blocks of length 764
| |
| +--Restrict columns:64 columns starting at column 164
| |
| +--Restrict to blocks of length 8
| |
| +--VBScript script Timeout: 10000 milliseconds
| 'Expand Mainframe EBCDIC Packed Decimal
| 'ie PIC S9(15) COMP-3 (an 8-byte field)
You can read more about handling mainframe reports in Working with Mainframe Reports
www.datamystic.com/docs
Splitting log files
Let's say you have a log file where you need to split entries into new files based on various
columns. In the example below, column 2 contains the event (EVENT FIRED or URL LAUNCH),
and column 3 contains the job number. We want all data from job 11 to be split into two files,
event_fired_11.log and url_launch_11.log.
18:33:39,EVENT FIRED,11,c:\program files\textpipe\textpipe.exe 1
18:33:40,URL LAUNCH,9,http://www.datamystic.com
18:33:39,EVENT FIRED,11,c:\program files\textpipe\textpipe.exe 2
18:33:40,URL LAUNCH,111,http://www.datamystic.com
18:33:39,EVENT FIRED,7,c:\program files\textpipe\textpipe.exe
18:33:40,URL LAUNCH,11,http://www.datamystic.com/1
18:33:39,EVENT FIRED,6,c:\program files\textpipe\textpipe.exe
18:33:40,URL LAUNCH,5,http://www.datamystic.com
18:33:39,EVENT FIRED,3,c:\program files\textpipe\textpipe.exe
18:33:40,URL LAUNCH,11,http://www.datamystic.com/2
18:33:39,EVENT FIRED,11,c:\program files\textpipe\textpipe.exe 3
18:33:40,URL LAUNCH,11,http://www.datamystic.com/3
Let's see how we would do this:
PDF created with pdfFactory trial version www.pdffactory.com
www.datamystic.com TextPipe Restrictions Page 6
|--Restrict to lines matching [^[^,\r\n]*,[^,\r\n]*,11,]
| | [ ] Include line numbers
| | [ ] Include filename
| | [ ] Match case
| | [ ] Count matches
| | Pattern type: 0
| | Context before: 0
| | Context after: 0
| |
| |--Restrict to lines matching [EVENT FIRED]
| | | [ ] Include line numbers
| | | [ ] Include filename
| | | [X] Match case
| | | [ ] Count matches
| | | Pattern type: 0
| | | Context before: 0
| | | Context after: 0
| | |
| | |--Merge output to file c:\event_fired_11.log
| | |
| | +--Remove all
| |
| +--Restrict to lines matching [URL LAUNCH]
| | [ ] Include line numbers
| | [ ] Include filename
| | [X] Match case
| | [ ] Count matches
| | Pattern type: 0
| | Context before: 0
| | Context after: 0
| |
| |--Merge output to file c:\url_launch_11.log
| |
| +--Remove all
|
|--Comment...
| All remaining data drops through here, so ensure it doesn't
overwrite your original file
Here we use Secondary Output Filters (found in the Filters\Special Menu), to capture the
restricted set of lines. We follow each of these with a Remove All filter to ensure these lines don't
re-enter the pipe.
There are many more possibilities - this is just to get you started.
PDF created with pdfFactory trial version www.pdffactory.com
www.datamystic.com TextPipe Restrictions Page 7
Feedback and Questions
If you have feedback or questions about this documentation, please email us at:
support@datamystic.com
We can also send you updated sample filters from this article, or sample filters tailored to your
data processing needs.
More White Papers and Documentation like This
Available from:
www.datamystic.com/docs
TextPipe Pro Evaluation
You can download a free 30 day trial of TextPipe Pro from
www.datamystic.com/textpipe-wp.exe
You can also access our other downloads from
www.datamystic.com/freetrials.html
Please contact us if you have any questions, difficulties or queries.
Contact Details
DataMystic
5 Bond Street
Mt Waverley
Victoria 3149
Australia
Email: sales@datamystic.com
Web site: www.datamystic.com
Phone: +61-3 9913-0595
Fax: +61-3 8610-1234
PDF created with pdfFactory trial version www.pdffactory.com