• Support
  • Forums
  • Blogs

USM storage virtual vs hardware

rdiethrdieth

New Life Form
+6
can someone explain why there is such a discrepancy between the storage on the AV Hardware VS the VM? we are running into storage issues with almost all of our clients but when i checked the recommended specs, i noticed that the hardware specs state 1.8TB but on the VM is states 1TB only.

Share post:

Answers

  • Hello rdieth,

       The "USM Enterprise" (Hardware only) comprise of a larger HDD storage; the USM-Logger (Enterprise) consists of a 2.2Tb HDD (1.8Tb after formatting).   Our other USM appliances (AiO // Standard-Server // Standard-Sensor // Remote-Sensor ) have a template size of 1.2Tb, and generally after being partitioned and formatted, will be just shy of 1Tb.   

       When deploying a Vm, we recommend THICK provisioning the drive; as this prevents the USM from expanding the .VMDK and filling the Datastore.  Even after deleting the logs off a USM (or OSSIM), VmWare it does not shrink the .VMDK.  But, you can do this manually; provided by VmWare as a Knowledge Base article :: 



       I suspect that you if are running in to a 'storage issue', this is most likely due to improper retention. You can get a good idea of your "retention" with the following information. 



    Your Database stores (on average) 40M events per day. You can estimate your retention with :: 

       [EPS] x [60sec] x [60min] x [24hr] = Your Daily EPS. (EPD) 

       40,000,000 / [EPD] = Average Retention 


    Example :: 
       40/EPS * 60/sec * 60/min * 24/hours = ~3.5Million Events Per Day. 

       40million Events Database / ~3.5Million EPS = 11.42 days of retention. 


       "Retention" is defined as searchable SIEM data in "SIEM View" on your USM//OSSIM appliance. 



     ** This is not recommended (as it has been known to cause less than ideal effects) but is possible to expand the template of the VMDK. However, please keep in mind that the larger the HDD, the longer it will take for SIEM Searches, Reporting, and potentially overall Ui and correlation performance. **



       All the best,

    - kratos
  • Kratos thanks for the detailed response, i was actually referring to the AIO having 1.8TB not the enterprise server, look at this PDF on hardware specs https://www.alienvault.com/docs/data-sheets/AV-USM.pdf  

    unfortunately when we first started with alienvault we were told there was no problem with using thin provision when deploying the images. we learned the drawbacks the hard way and are in the process of converting to thick. my problem is that we have a couple clients that average 300 EPS with spikes in traffic up to 700 EPS that gives us only 1-2 days worth of retention time. i know some logs can be viewed as raw logs but the searchs are not as flexible and user friendly as the siem, also most reports run from the SQL database anyway


  • Hello rdieth,

       Apologies on the miscommunication; that would be correct on the 1.8Tb for the AiO appliances. Enterprise Logger contains 2.2Tb (dedicated storage). In regards to the "thick vs. thin" provisioning, it is entirely acceptable to thin provision; the only caveat being that if the HDD does not have enough storage to begin with, it could expand the .VMDK and fill the HDD.  

       With the spikes in EPS, in order to expand the amount of searchable // reportable data, our supported solution would be the deployment of a Logger appliance. Alternatively, you can attempt to #scp the files to a warm//cold storage medium, and when needed (as long as there is free space on the USM), you could #scp them back.  Albeit, it may not be the most viable recommendation, but for a security admin on a budget (not looking for the additional Logger), it provides a free option, with the tradeoff of a little extra work with moving the logs.   

      Please let me know if this helps. 

    All the best!

    - kratos
  • then do you know why the AIO appliance has 1.8TB and the AIO vm is templated for only 1TB even though they are suppose to be the same device? what was the reason behind the design decision?

    we have already deployed loggers to our larger clients with storage issues. while that did prevent us from running out of space on the hard drive it doesnt resolve the issue of being able to run proper reports, the amount of data the SQL database keeps does not change.

    for example without accounting for spikes in traffic, one of our clients averages 500 EPS, thats 43 million events per day which is 3 million events over the default 40 Million event limit before the logs are compressed and archived. that is only 1 days worth of events in the SQL database, we actually had to raise the retention limit from 40 to 70 Million which only gave us 1.5 days worth of logs. we can raise it higher but its been highly frowned upon by AV for obvious reasons such as exponentially longer searchs in the SIEM. we then attempted to run reports using the raw logs but found out that alot of the reports dont even work because of the way there designed to pull reports. the plugins can on have one product type association but firewalls can also be a vpn device

    going a step further the AIO has an EPS limit of 1000, 2500 if you disable the sensor. if there was a client that averaged 1500 EPS thats 129 Million EPD what would be the solution then?
  • Hello rdieth,

       Do you know if you deployed a Standard Server, vs an AiO server?  

       In regards to the amount of EPS and Retention, you are more than welcome up increase the number of millions of events to store in the database; the only caveat is that you may run the risk of performance decreasing and/or filling the HDD. The reason being, is that you are not expanding the amount of MySQL storage on the USM. With a larger database, it will take longer to run searchings and reports. 

       If you disable the "sensor" profile (disabling NIDS and all plugins) on the AiO, in increases the amount of EPS from 1,000 correlated events to 5,000 correlated events. A standard sensor has the ability to collect 2,500 EPS.  If you have an appliance that is pushing 129million EventsPerDay, you could look at moving the logs to another storage medium, and then moving them back in the event that you ever need to investigate. 


          * Backing Up and Restoring Raw Logs ::
    https://linkprotect.cudasvc.com/url?a=https://www.alienvault.com/documentation/usm-appliance/usm-backup-restore/raw-logs-backup-restore.htm%3fHighlight%3drsync&c=E,1,KRpXsPvLqlhf6WYhv7oqeLn7hOxnLsC1UG37xDTLmwHgFsYlsHqhi-duRaj6Yh-NDTyXLJ54k00BbB5pUDssP39PLwxuPkBpah_y4gDrvF7l-vD1wxel&typo=1

       * Archiving Raw Logs to Free Up Space on USM Appliance All-in-One or USM Appliance Logger ::
    https://linkprotect.cudasvc.com/url?a=https://www.alienvault.com/documentation/usm-appliance/kb/2015/08/archiving-raw-logs-to-free-up-space-on-usm-aio-or-usm-logger.htm%3fHighlight%3drsync&c=E,1,6IAI4hV6QLhEv_S0lDd4AnedmitFfwQpyvc8BPaNDsq5m-xQj0BPFs2ac1tHX0UMvpi7_JtDne3C9dzBG0tOUS_kuJ8JGIfZkNP1FnNSZKVg5Lui8fH4GA,,&typo=1


       As it appears as though you have USM (instead of OSSIM), you can open up a case, and they would be able to move it to a Sales Engineer // Solutions Architect to have them answer your questions in more detail. They should also be able to scope your deployment to your particular environment. 

       Regards,

    - kratos
  •  "The reason being, is that you are not expanding the amount of MySQL storage on the USM." did you mean to say that you "ARE" expanding the the MySQL storage? or am i extremely confused?

    i have a ticket open with Colin Scott i will follow-up with him, i guess my last question is from a security standpoint what would the bare minimum amount of days worth of events to keep in the live SIEM? i understand the more events i keep the longer searchs take to complete.
Sign In or Register to comment.