[plug] grep -v on subnets

[Brad Campbell]

G'day Thomas,

Appreciate that, it has been most helpful and gave me a few more tools to use.

Regards,
Brad

On 15/3/22 23:49, Thomas Cuthbert wrote:
> I use rgxg on a daily basis for grepping firewall rules and logs. Before that I used grepcidr but didn't like how it was a standalone grep tool. rgxg just generates the regex string so you can use it in sed/awk/grep. Anything more advanced I'd look at tokenizing addresses into arrays or use perl -e/python -c standard ip modules.
> 
> If you want to get a list of blocks owned by an organisation you can query Arins whois service or query an internet routing registry like radb.
> 
> https://rgxg.github.io/ <https://rgxg.github.io/>
> https://www.radb.net/support/tutorials/how-to-query-merit-radb.html <https://www.radb.net/support/tutorials/how-to-query-merit-radb.html>
> 
> 
> On Sat, 12 Mar 2022, 9:47 am Brad Campbell, <brad at fnarfbargle.com <mailto:brad at fnarfbargle.com>> wrote:
> 
>     G'day all,
> 
>     I've knocked up a simple log processor in bash to dump ip addresses that access our zimbra server on a daily basis.
> 
>     It's not pretty but it works :
> 
>     LIST=`zcat $LOGNOW | \
>             grep -o 'oip=[^;]*' | \
>             sed 's/oip=//g' | \
>             sort | \
>             uniq | \
>             egrep -v '(^192.168.|^10.8.)' `
> 
>     At the moment it gives me a list like :
>     1.126.106.244
>     1.126.109.132
>     1.146.128.86
>     1.146.135.178
>     1.146.167.203
>     1.146.169.253
>     1.146.174.225
>     1.146.175.139
> 
>     That is yesterdays list and are all Telstra mobile CGNAT addresses. I'd like to be able to filter those based on the known telstra subnets and just leave the outliers (like the continuous stream of Russian bots hitting the EWS port scanning for Exchange vulnerabilities)
> 
>     What I'd like to be able to do is replace the final egrep with something that can handle subnets (and a list of them), for example : 1.128.0.0/11 <http://1.128.0.0/11>, 1.120.0.0/13 <http://1.120.0.0/13>, 192.168.0.0/16 <http://192.168.0.0/16>, 10.0.0.0/8 <http://10.0.0.0/8>
> 
>     Before I converted the whole thing to python and implemented subnet filtering I thought I'd ask and see if anyone has something clever they've used/seen.
> 
>     Regards,
>     Brad
>     _______________________________________________
>     PLUG discussion list: plug at plug.org.au <mailto:plug at plug.org.au>
>     http://lists.plug.org.au/mailman/listinfo/plug <http://lists.plug.org.au/mailman/listinfo/plug>
>     Committee e-mail: committee at plug.org.au <mailto:committee at plug.org.au>
>     PLUG Membership: http://www.plug.org.au/membership <http://www.plug.org.au/membership>
>