What to do? Is cloud is the option?? Is that possible to analyze the data and message computing using Cloud? There is a solution with Microsoft Codename “Cloud Numerics” it is easy to write models and applications and can handle the large datasets. You can use visual studio to write these types of applications.
Create a project of type Cloud Numerics Application in Visual Studio. You immediately have a access to an extensive set of numericals functions ranging from basic mathematics to statistics and algebra.
All the complex numerical analysis can be done with .NET Cloud Numerics programming framework. All you need to do is make sure that your large data set defined as distributed array.
After that you just need to apply mean and Matrixmultiply operations
You can develop your model or application on your desktop using small data set. When you ready to scale out your application to cloud simply change your small data set reference in your code to large data set.
Now you can use the Cloud Numerics deployment wizard for deployment to the cloud.
Now you are running your big analytics on cloud. Cloud managed to provide ability to run complex data models on cloud. This enables you to process and analysis much larger datasets.
More about cloud numerics can be read here