Big Data - not just for Big Companies

In the last week, have you made journeys with a switched on mobile phone, posted, liked or uploaded a photo on Facebook? Have you searched the internet? Watched videos on Youtube?

Then you are data, a small part of big data in fact.

Does your company have a website? Does it do any marketing or sales?

Then it’s generating big data, the hundreds or thousands of data points that define any one journey through a website quickly accumulate.

What data exists about your company outside of your firewall, and how this could correlate to your internal business operations? For example product reviews, discussions, web articles… do they discuss concepts that are also contained within your production & CRM systems?

And what can we do with this goldmine of information? Well it’s different for every company, and on first thoughts you may think big data means big budgets and big projects and therefore restricted to big companies, but this really isn’t the case.

To quote the Cambridge Institute of Manufacturing, “If you are an established SME and not thinking about these issues, it's time to jump on the Big Data train, for if you don’t the danger is you’ll be left sitting in the sidings while newer start-ups and bigger players pull out of the station and power ahead.”

Businesses are understanding how they are perceived by their customers, sensing demand in time to fulfil it, tracking performance of their products through sensors and detecting fraud through transaction correlation, whether it’s a vehicle rental company being aware of and addressing a Facebook storm among its customers, an online retailer tracking spikes in sales for unexpected products due to current global events and proactive pushing similar products, or a tractor manufacturer notifying its customers of potential mechanical problems before they happen

It’s a new age of communication we’re living in. Individuals are starting to understand that their thoughts, movements and behaviours are being stored as a collection of ones and zeros. Machines need to get used to being monitored more carefully and maintained before they break down. And of organisations need to understand both the opportunity and responsibility this big data presents. Enjoy responsibly!

Written by Angus Menter, BI Practice Manager, DSCallards

Case Insensitive Lookups with SSIS

Whilst recently compiling a reasonably complex transformation to bring in some data from an Excel spreadsheet I was left in the situation where I had a some look ups that were failing to resolve due to case sensitivity. This was a failure on my part really as I should have understood from the outset that this could have been the case and I should have factored this in from the start but I didn’t. No problem usually, You can just change the case at the source and and in the look-up component and you’re there. However… In my situation this was not feasible without me throwing myself out of the nearest window and making finger paintings with my own blood in my dying moments. The reason for this is that I had about 50 separate sources which is an onerous task in itself. In addition to this as I was ‘unioning’ all of this data together using the union component and then going through several more complex transformations with additional unions along the way it was getting very messy, The union components do not refresh with new fields when the source changes and so the only way to correct this is to drag new union components on in place of the old ones! This in turn means that all of the adjoining components also need to be refreshed and even talking about it I can feel the end approaching.

Read the full blog here. 



Written by Conrad Rowlands, Team Leader and Developer, DSCallards 

Happy to do Superficial ...

So…. Sometimes the very simplest of things can take gargantuan efforts to work out and you find out that the detail really is just too much detail.  

Just such a  situation happened with me this week when working on a site I am developing. I needed to implement the sending of an email once a post had been received from an MVC page to indicate that a booking had been made. The mail would be sent both to the establishment where the booking was made as well as a copy being sent to the person making the booking. I elected to use the inbuilt System.Net.Mail class, how hard can it be right?…… Well A damn sight harder than I thought that’s for sure! This week there will be no code sample as code was in this instance not at fault or has no bearing on what was happening. To cut a very long and very boring story short it did not matter what I did I could not get the mail class to send the mail; upon sending, the operation would timeout or the server would disconnect etc etc. It made no sense so I took a step back. I had been utilising my Yahoo account using the smtp.mail.yahoo.com server and of course as this was secure and requires authentication I used port 465. 

So I thought maybe the issue is with Yahoo and I changed to my Google account instead using the smtp.googlemail.com SMTP server again with port 465. Still no dice…..Odd. Much headbanging later and I stumbled across this StackOverflow article which led me on to this blog. 

Now I’m not going to lie and say that it all makes sense to me. It has at least allowed me to at least find a workaround by in this instance using port 587 on my Google SMTP server. If I sat down and thought about it I’m sure I could understand in finite detail what exactly Explicit SSL is and why the .NET mail client only supports this type of SSL, the point is…. I just don’t want to! I feel like I’ve already wasted enough time dealing with this bug (I’m sorry, Its not a feature its a bug whatever they say and one they seem reticent to fix). 

Normally I’m a man who likes to get the detail, a superficial overview is never normally enough as it leads to design mistakes. However in this instance I now know in enough detail for me that the .NET mail class does not support all SSL SMTP conversations and I’ll continue to bask in my own ignorance.

Written by Conrad Rowlands, Team Leader and Developer, DSCallards