All Questions
Tagged with command-line text-processing
867
questions
-1
votes
0
answers
50
views
Process only 2nd line from the utility output
The output of command line utility ./Utility -P consist of two rows of text.
I use shell script to capture standard output from this sub shell and show it as notification use zenity:
#!/bin/bash
...
- 557
2
votes
1
answer
37
views
How to get package version in one command line in bash [duplicate]
I'm trying to get the installed version of systemd in the following command, but it returns all lines contain the keyword "systemd".
# dpkg -l | grep " systemd "
ii gnome-logs ...
- 393
-1
votes
1
answer
68
views
How do I find files of a certain extension(s) except in paths that contain certain strings and then print the result to a file?
Upon a little search, I came up with this command:
> find . -type f \( -name "*.exe" \) | egrep -v
> 'games_1|rei|games|tor|jogos|mods|text' > 1
What I wanted to do was find files ...
- 422
6
votes
5
answers
1k
views
Cut folder names out of string
How do I extract the folder named test from the following string?
TRUE|/home/linux/test|(null)|
- 79
0
votes
4
answers
814
views
How to use sed to remove newlines above and below a string?
I'd like to change all occurrences of \ntoken1\n and \ntoken2\n to token1 and token2 in a file).
I thought this would do it:
${
N
N
s/\ntoken1\n/token1/
s/\ntoken2\n/token2/
}
but it does not.
Thanks ...
- 43
0
votes
2
answers
335
views
Removing spaces between words in a text file using a bash script
I am using a bash script that parses data from a bunch of different text files and generates a .dts file. The .dts file contains spaces between each column title and I need to remove them as it makes ...
- 11
2
votes
3
answers
638
views
What grep pattern allows to print only one-word lines?
I want to print the lines with only one no-spaces subsequences of characters while use grep on the text.
title after grepping should to print title
the sentence after grepping should not print ...
- 51
0
votes
3
answers
168
views
sort in shell using multiple -t
I have a file that contains:
192.168.130.175 2014-09-04 10:25:01 /index.html
192.168.138.244 2014-09-04 11:23:00 /index.html
192.168.138.244 2014-09-04 10:29:37 /...
- 3
4
votes
2
answers
1k
views
Using grep to search for line that begins with a variable whose value is a dollar sign
I found a lot of very similar problems, but not exactly this.
I have a text file with the following contents (no repeats, fixed 4 characters per line):
A1234
G1234
$1234
M1234
I'm trying to use grep ...
- 43
0
votes
3
answers
229
views
Filter RE in grep
The command that I have is:
grep -E '^ [0..9]'
The pattern that I have
24
-38.54270131
24
-38.54085331
24
-38.53603119
24
-38.53427780
24
-38.53396521
24
-38....
- 352
0
votes
3
answers
212
views
Pull mac address via bash script
A bit background.
Im creating a script that will run on USB stick with Ubuntu. It needs to pull various info from the current laptop its connected to and save it to a file.
Right now Im using "...
- 3
2
votes
4
answers
657
views
using bash, create files based on pattern match
we have a file with content as below.
/* ----------------- pattern_1 ----------------- */
jb: pattern_1 Data: asdflj adfas Log: dir/log1
/* ----------------- pattern_2 ----------------- */
jb: ...
- 23
0
votes
1
answer
152
views
Search for a character string in Html files and output this and the <title>.*</title> tag of the file
I want to search Html files for a character string across all lines. If the string is found, then this string and the <title>.*</title> tag of the file shall be output. The search must be ...
- 33
1
vote
1
answer
248
views
Deleting lines beginning with a string
I wrote a bash script that I use to backup certain directories to the cloud. I then direct that output to a log file.
The log file consists of lines starting with either "upload" or "...
- 51
4
votes
2
answers
537
views
Modify text file with command
How can I modify a text file to do the following in 1 command?
I have a text file called Errors.txt with the following 3 lines:
Error in File /Test1/A.png
Error in File /Test2/B.jpg
Error in File /...
- 51
1
vote
1
answer
237
views
How to iterate eml file with mailutils?
How would a directory of .eml files be iterated?
nicholas@mordor:~$
nicholas@mordor:~$ mail -f foomail/foo.eml
"/home/nicholas/foomail/foo.eml": 0 messages
nicholas@mordor:~$
Whereas, I'...
- 1,234
-1
votes
1
answer
299
views
How to import an mbox file into the Pine or Alpine mail client?
I'm a light Alpine e-mail client user.
If I navigate into ~/mail and run mail -f baz I'll get an expected list of old alpine/pine messages.
Alpine doesn't use mbox.
I added an mbox backup to that ...
- 1,234
-1
votes
1
answer
175
views
Bash script to compare lines in 2 formatted files
I have two formatted files with lines with fields separeted by a delimitter ( in this case, '*' )
File1:
qwerty*asd*1111111*poiu*222222*mnb
nbcvxm*hjf*3333333*jhjd*444444*bvc
dhfkjs*bmb*5555555*vcxz*...
user1746775
5
votes
6
answers
2k
views
Delete all lines after a certain number of lines
How can I truncate after 1000 lines in a csv file? The goal is to keep only the first 1000 lines in the file and delete all others.
- 487
0
votes
1
answer
239
views
Need to create csv file from content of another file
I need help in writing a command or shell script that generates a csv file as per below details -
My file has content like -
+ Creates a new NotificationEventMapping
POST https://sbc.com:1234/openapi-...
- 11
1
vote
2
answers
309
views
Process tesseract output: remove line breaks, concatenate individual pages
I have a pdf that I cannot directly process into .txt to go through piper TTS because the output from the .pdf is missing letters and otherwise generally unintelligible (https://pastebin.com/J3rASD2F):...
1
vote
1
answer
125
views
Sorting a text file based on numerical values present in each line
I have a text file containing paths of various wav files. But when they are created they are sorted based on alphabetical order. Hence the paths are like this
/home/wav_path/...
- 143
-1
votes
2
answers
337
views
How can I make output of sed command as variable?
How can I make the below command as variable ? I mean I need to assign this command to variable to use it within for loop.
for i in $(sed 's|/home/bssuser/Scripts/Amany/log/bssuser/CDR/...
0
votes
1
answer
37
views
Join details from two lines based on a match of the first two or three fields
I'm seeking help to extract / merge data from a single csv file.
The file contains the arrival and departure times of ships over a few months. The details of the arrival and departure are on separate ...
- 11
0
votes
1
answer
74
views
How to find a string in files, said string including line breaks, ", @
I know grep allows me to search for a string in files, however I did not manage to find the right method to search for the following string:
"snapshot3" : {
"@custom_name" : ...
- 169
0
votes
0
answers
149
views
How to get some lines before and after in log files using tail and grep? [duplicate]
How to get some lines before and after in log files using tail and grep ?
0
votes
1
answer
152
views
How can I determine the prematch and postmatch using egrep or ksh under linux
I am working on a issue at the office running on a linux system.
I would like to be able to either use egrep or ksh pattern matching to determine not only the matched string but I also need to be able ...
- 103
0
votes
1
answer
186
views
How to delete multiline string with grep
Let's say I have file.txt with following content (simplified for example. In real it's much more complex):
This line I DON'T want to remove \
common substring
This line I want to remove \
...
- 123
2
votes
1
answer
178
views
Multiply found text in sed - is it possible?
This is what I have:
echo -e "a\t4\nb\t7\nc\t2\nd\t12\n" | sed -rn 's/(.*)\t([0-9]*)$/\2\t\1\1\1\t/p'
4 aaa
7 bbb
2 ccc
12 ddd
This I want:
4 aaaa
7 bbbbbbb
2 cc
...
- 1,030
0
votes
1
answer
419
views
Add a list to specific lines of a (YAML) text file using python
I have an Ansible module file for NetBox with this content:
- name: "Test NetBox modules"
connection: local
hosts: localhost
gather_facts: False
tasks:
- name: Create device ...
- 9
1
vote
5
answers
387
views
list items correctly in bash with a space in their name
I have for example following Files:
test1.txt
test2.txt
test space.txt
I use the following command:
output=$(ls -l --time-style=long-iso | awk '{print $8, $6}')
And I get following output:
test1.txt ...
- 19
2
votes
2
answers
4k
views
How to know a JSON array is an empty array in bash?
I use this code to consume GitHub API and automate some tasks:
curl --silent -H 'Authorization: token github_access_token' 'https://api.github.com/orgs/OrganizationName/repos?per_page=100'
Sometimes ...
- 825
3
votes
2
answers
794
views
bash script to compare fields of two files and replace columns
I have two files.
File A:
newname,1,string
newname,2,string
newname,3,string
...
name,65500,string
File B;
oldname,1,string
oldname,2,string
oldname,3,string
...
oldname,65500,string
I need a bash ...
8
votes
7
answers
1k
views
Compare new txt file with old txt file and remove all data that matches
I have a new file with the following data separated by a carriage return
a
a
b
c
d
d
I have an old file that is also separated by a carriage return
b
d
How do I remove b & d from the new file ...
0
votes
2
answers
392
views
Shell or bash script to open multiple URLs saved in a .TXT file in a browser from the terminal?
Is there a shell or bash script to open multiple URLs saved in a .TXT file in a browser from the terminal?
I know that can be done with "firefox $(cat url.txt)", but:
I need something that ...
- 113
5
votes
3
answers
1k
views
how can I get unique ip addresses from /var/log/auth.log?
How do I only show the unique IP addresses in /var/log/auth.log ?
I have a server getting hit constantly, and while I have set up fail2ban, the requests keep coming. So I started writing to ISPs to ...
- 14.9k
0
votes
2
answers
587
views
How to parse large files looking for a word
I have some extremely large json files
I am not even really sure if some are even written correctly
Therefore viewing the with a editor is not possible (it freezes vscode etc)
I can view them with ...
- 475
1
vote
1
answer
335
views
How to find and replace a string in all files on ubuntu including root files with another string
I am running Ubuntu 20.04 server. It is a fully configured LEMP email and web server. In my super long process of building this, i know I entered my servers IP address on one or many of my ...
- 113
0
votes
2
answers
609
views
print multi-line text with spaces aligned on the first line
Issue:
I write relatively complex functions where I have to make up the code with spaces.
It is expected that the first line with the content (non-space character) will be determined, N spaces will be ...
- 41
0
votes
1
answer
1k
views
how can i find the average of data in a column in a txt file?
I need to write a script that can find the average of numbers in a column and then print out the value when run. I am quite new to Linux commands, and I've tried my best, but I don't even know how to ...
5
votes
2
answers
815
views
How to add double quotes around string and number pattern?
Hi I need to add double quotes in a pattern in 300k lines. I'm trying to use sed and I read multiple inquiries here and other sources, but I can't seem to understand its syntax.
I have:
chr1 ...
- 51
5
votes
1
answer
491
views
combine single text with multiple lines of file
I am trying to combine work tvg-name with multiple lines of a file
tr '\n' '~' < file1.txt | sed s/~/tvg-name/g
results in only first single-line print
tvg-namegama1
my file1.txt has more than ...
- 53
1
vote
2
answers
137
views
Unique values separated by semicolons (thumbnailers)
The file /usr/share/thumbnailers/totem.thumbnailer contains:
[Thumbnailer Entry]
TryExec=/usr/bin/totem-video-thumbnailer
Exec=/usr/bin/totem-video-thumbnailer -s %s %u %o
MimeType=application/mxf;...
- 287
-3
votes
1
answer
49
views
cut even lines and merge to upper line
I have a file like with CRLF line terminators:
Green Day: Rock Band Xbox 360
3074
Green Hell Steam
27015-27030, 27036-27037
Green Hell Switch
6667, 12400, 28910, 29900, 29901, 29920
Adios Steam
Adios ...
1
vote
3
answers
62
views
extract colname
I need to get colnames but just want one colname in each line so could you please help how to change below command.
head -n 1 betas_1.csv >col.names
- 167
2
votes
2
answers
538
views
bash script to append file's name at the end of each line in it's file
I have some files in a directory:
a.csv b.csv c.csv
I wanted to write a bash script to add a filename at the end of each line of its file, like:
a.csv
line1 ,a
line2, a
line, a
b.csv
line1, b
line2, ...
1
vote
1
answer
2k
views
appending variable to end of file with sed
I have dates on the last line in files like this:
20230220
and I want to add hours like this:
2023022000
The hours are continuously assigned to a variable i in a loop.
How to use sed inside the loop ...
- 11
1
vote
1
answer
460
views
grep and awk commands not returning or storing output to the variable
I have a (users.csv) file containing a list of id's of all employees. I need is to loop through this file for each employee, search if it exists in backup.txt file using grep and store the output in a ...
1
vote
3
answers
2k
views
awk or sed command to replace line break plus text containing spaces
An answer to another question suggests sed -i 's/original/replacement/g' file.txt to replace specific words in a text file. My starting situation looks like this:
Item: PRF
Type: File
...
- 340
0
votes
0
answers
216
views
Tail in combination with sort
I use tail in a variable to echo contents of multiple files in a folder.
VAR="$(tail -n +1 DIR/*/file)"
This outputs all the files contents with the needed filename on top like
==> /...
- 111