Big Data - not just for Big Companies

In the last week, have you made journeys with a switched on mobile phone, posted, liked or uploaded a photo on Facebook? Have you searched the internet? Watched videos on Youtube?

Then you are data, a small part of big data in fact.

Does your company have a website? Does it do any marketing or sales?

Then it’s generating big data, the hundreds or thousands of data points that define any one journey through a website quickly accumulate.

What data exists about your company outside of your firewall, and how this could correlate to your internal business operations? For example product reviews, discussions, web articles… do they discuss concepts that are also contained within your production & CRM systems?

And what can we do with this goldmine of information? Well it’s different for every company, and on first thoughts you may think big data means big budgets and big projects and therefore restricted to big companies, but this really isn’t the case.

To quote the Cambridge Institute of Manufacturing, “If you are an established SME and not thinking about these issues, it's time to jump on the Big Data train, for if you don’t the danger is you’ll be left sitting in the sidings while newer start-ups and bigger players pull out of the station and power ahead.”

Businesses are understanding how they are perceived by their customers, sensing demand in time to fulfil it, tracking performance of their products through sensors and detecting fraud through transaction correlation, whether it’s a vehicle rental company being aware of and addressing a Facebook storm among its customers, an online retailer tracking spikes in sales for unexpected products due to current global events and proactive pushing similar products, or a tractor manufacturer notifying its customers of potential mechanical problems before they happen

It’s a new age of communication we’re living in. Individuals are starting to understand that their thoughts, movements and behaviours are being stored as a collection of ones and zeros. Machines need to get used to being monitored more carefully and maintained before they break down. And of organisations need to understand both the opportunity and responsibility this big data presents. Enjoy responsibly!

Written by Angus Menter, BI Practice Manager, DSCallards

Case Insensitive Lookups with SSIS

Whilst recently compiling a reasonably complex transformation to bring in some data from an Excel spreadsheet I was left in the situation where I had a some look ups that were failing to resolve due to case sensitivity. This was a failure on my part really as I should have understood from the outset that this could have been the case and I should have factored this in from the start but I didn’t. No problem usually, You can just change the case at the source and and in the look-up component and you’re there. However… In my situation this was not feasible without me throwing myself out of the nearest window and making finger paintings with my own blood in my dying moments. The reason for this is that I had about 50 separate sources which is an onerous task in itself. In addition to this as I was ‘unioning’ all of this data together using the union component and then going through several more complex transformations with additional unions along the way it was getting very messy, The union components do not refresh with new fields when the source changes and so the only way to correct this is to drag new union components on in place of the old ones! This in turn means that all of the adjoining components also need to be refreshed and even talking about it I can feel the end approaching.

Read the full blog here. 



Written by Conrad Rowlands, Team Leader and Developer, DSCallards 

Happy to do Superficial ...

So…. Sometimes the very simplest of things can take gargantuan efforts to work out and you find out that the detail really is just too much detail.  

Just such a  situation happened with me this week when working on a site I am developing. I needed to implement the sending of an email once a post had been received from an MVC page to indicate that a booking had been made. The mail would be sent both to the establishment where the booking was made as well as a copy being sent to the person making the booking. I elected to use the inbuilt System.Net.Mail class, how hard can it be right?…… Well A damn sight harder than I thought that’s for sure! This week there will be no code sample as code was in this instance not at fault or has no bearing on what was happening. To cut a very long and very boring story short it did not matter what I did I could not get the mail class to send the mail; upon sending, the operation would timeout or the server would disconnect etc etc. It made no sense so I took a step back. I had been utilising my Yahoo account using the smtp.mail.yahoo.com server and of course as this was secure and requires authentication I used port 465. 

So I thought maybe the issue is with Yahoo and I changed to my Google account instead using the smtp.googlemail.com SMTP server again with port 465. Still no dice…..Odd. Much headbanging later and I stumbled across this StackOverflow article which led me on to this blog. 

Now I’m not going to lie and say that it all makes sense to me. It has at least allowed me to at least find a workaround by in this instance using port 587 on my Google SMTP server. If I sat down and thought about it I’m sure I could understand in finite detail what exactly Explicit SSL is and why the .NET mail client only supports this type of SSL, the point is…. I just don’t want to! I feel like I’ve already wasted enough time dealing with this bug (I’m sorry, Its not a feature its a bug whatever they say and one they seem reticent to fix). 

Normally I’m a man who likes to get the detail, a superficial overview is never normally enough as it leads to design mistakes. However in this instance I now know in enough detail for me that the .NET mail class does not support all SSL SMTP conversations and I’ll continue to bask in my own ignorance.

Written by Conrad Rowlands, Team Leader and Developer, DSCallards 

Supporting PHP in Visual Studio

I was very recently asked to lend my support for maintaining a website we have which is written in PHP. Now normally I’d run straight off to my trusty little 2010 macbook (which, by the way, has NOT degraded in speed in all that time nor does it require constant reboots..do you hear that windows? I digress) and use any one of the free or cheap plethora of tools to help with this task. Personally I tend to use Aptana Studio. Just for once I thought I would try and leverage my normal windows work machine for this task and so I turned straight to Visual Studio. It turns out that there is an add-on for studio made by the guys at Devsense. So using the Visual Studio Extensions and Updates manager I installed the add in called ‘PHP Tools for VS 2013′ (which is on a 30 day trial) and awaiting the amazingness…..

Time passed and true amazingness did not happen and so I decided that I would just get on with my work instead.

I downloaded the source for the website put it onto my machine and then started the process. As the project is an already existing site i needed to run the ‘Project From Existing Code’ menu item available from the File/New menu item. This takes you through a simple wizard and at the end of this process you have a visual studio project with all of your PHP code. Great, so now down to it…. Unfortunately what I did not realise was that this site made use of PEAR extensions, thus everytime i tried to run a page with email capabilities it would just break unable to resolve the necessary libraries.

Click here to read full blog.

Written by Conrad Rowlands, Team Leader and Developer, DSCallards

Augmented Reality

So, IBeacon…. You may or may not have heard of this technology, Indeed up until a year or so ago I must confess that I had not. These funny little items then ended up on my desk, 3 in all, and I wondered what to do with them. As with most things that cross my desk eating them was an option but they looked pretty inedible so after an interval of some months I instead opted to find out what I was supposed to do with them. So i headed over to the Estimote website who are the manufacturers.

 “Estimote Beacons and Stickers are small wireless sensors that you can attach to any location or object. They broadcast tiny radio signals which your smartphone can receive and interpret, unlocking micro-location and contextual awareness.” Instantly I can imagine that sales and marketing teams are practically wetting themselves at yet another opportunity to push their unwanted wares upon us in yet more insidious ways. I know that the first and every other subsequent app that tries to promote their wares to me in this way will be removed from my life for ever. I do not need any help in spending money. However looking beyond the more obvious tired use case for these beacons and examining the phrase ‘contextual awareness’ raises some real and positive use cases that I as an individual would actually subscribe to. ‘Contextual awareness’ or ‘Augmented Reality’  is the process of providing supplemental information about an environment or area that would further help the user to understand or interact with his/her surroundings; that is, enriching the real world with appropriate digital information ensuring that that ‘viewer’ has access to a fuller understanding of the environment which they find themselves in. So real world examples…. Well mostly people are trying to sell you stuff.. sigh, you know the drill, you walk into a department store and as you approach the perfume department the stores app will sense your proximity (and odour!?!) and will tell you what special offers are available today.

Click here to read the full blog.

Written by Conrad Rowlands, Team Leader and Developer, DSCallards

Blogging: Lead By Example

“If you want it enough you’ll make time.”

It’s a phrase I use often in meetings or one-to-one  situations when someone is letting me know how they are super busy and can’t fit “it” into their life.  I’m not different I didn’t ring my mum again this week because I was just too busy.

Of course that’s absolutely nonsense. I chose not to ring her for one of many reasons, all utterly plausible and all legit but the one reason that was not plausible or legit was that I didn’t have time. When I was waiting for the pasta to boil, instead of watching half an hour of the x factor, while waiting to pick my daughter up from yet another dancing lesson all time that could have been better used … but I chose not to.

And so it is with writing a blog. In our office there is a notion from the hierarchy that content is king. I am part of that hierarchy and believe that content is king passionately. Why is Google Google because it has content – lots of it and with it they have huge power. On a different level it’s still the same, if you write a good blog and put that content out there in the ether you see how many people read it -  good content eventually finds its way round and for those that it is addressing it’s important!

I work with some prolific blog writers, mainly technical people who use others’ ramblings to help themselves so feel at one writing articles in the hope that they are giving something back into the environment they take from. However out of that techie-to-techie world getting staff to write blogs or content becomes harder. The Sales Team have come up with some excellent ideas of helping others, answering the questions they had when they started, but their second or third blog becomes harder.  The Administration staff struggle with “who’ll read my blog  anyway” syndrome which is strange because when questioned they all accept they read blogs to help them with day-to-day activities.

Then there is management, it becomes a mix of all the above.

They are not technical enough to write technical to technical.

They are too savvy to write the “Beginner’s Sales” or “How To Guide”.

They are not specialist enough to write “The Top 10 Social Media Skills” or “Top 15 One-to-One Questions”.

They wonder who would read their solutions, help, problems, ideas anyway.

And of course … managers don’t have time!

Written by Adrian Handley, Managing Director, DSCallards

Ten Reasons Why I Hate Blogging - Part One

I am not a person who is enveloped by the ubiquity of social media. I am on Facebook, Twitter and LinkedIn, but I tend to treat them as ‘read-only’ most of the time, using them to catch up on the latest news when I have the time to do so, but rarely choosing to post anything.

Relevance – this is what I struggle with. I try to think about the reader of my potential output of social media and think – is it relevant for them? Is it appropriate?

More often than not, this perpetual self-analysis and self-deprecation tends to lead to convincing myself not to bother.

Blogging, on the other hand, is a different matter. I understand that this has many important uses – a fountain of knowledge for some, or an aide memoire for another.

But again, the relevance question always comes back to haunt me – does anyone ACTUALLY care what I think?

I am not yet convinced, but if you, like me, are currently being “persuaded” to write more blogs, here are five things that I find difficult which serve as reasons for hating the process:

1.    Starting The Blog

This is always the first stumbling block for me. How should I start? The pressure to engage your audience whilst being witty, intelligent and articulate and at the same time drawing your readers in for the long haul is always the first big obstacle to my blogging.

2.    A Clear Message
 
Those of you who know me, and those of you who have battled through my blog this far, will know that my writing style is somewhat circumlocutionary!  Therefore, finding a clear message which readers can take away, with a warm feeling of being informed and inspired, is not something that comes easily to me. I also often find that this contributes to the first reason why I hate blogging – it’s hard to come up with a charismatic audience-gripping intro when you aren’t sure what you’re going to be talking about!

3.    Losing Momentum

So after getting your readers hooked with an opening paragraph worthy of a Royal Variety performance (see reason 1), the next big problem is keeping that momentum going whilst staying “on message” (see reason 2). I personally find this very hard, not being a natural writer; I tend to proceed by leading the interlocutor on a long and winding ramble into the wilderness of ambiguity, not knowing whether they are coming or going. Much like you are experiencing now, I imagine!

4.    Being Judged

Being naturally self-conscious, I am always concerned with blogs that the three people who do eventually happen upon my less-than-scintillating explosion into the blogosphere will be the ‘Simon Cowells’ of social media. Going through the process is painful enough, but the thought of it being read and slated (privately or publicly) by a ‘Blogging Baron’ is normally the icing on the Blog-hating Belgian Bun.

5.    The Big Finish

So – you’ve made it this far. Knocked them out of the park with your charismatic intro, a message that’s as clear a mud which you’ve stuck to like glue with all the momentum of a concrete elephant (mixed metaphors, anyone?) and dazzled the critics so far.

Now to round it off with a simple, concise yet informative conclusion that will resonate with readers and get tongues wagging around the globe.

I have to say that of the first five items, this is the most difficult for me. Concise is, as you now know, not normally in my vocabulary (why use a sentence when a paragraph will do just as well?) It’s also hard to conclude something without repeating yourself and sounding like you’re just filling up the word count. On top of the fact that most of your readers have probably now got more interest in the advert for Game of Thrones that is now scrolling across the foot of this blog site, it really is very difficult to round things off well.

I hope you (both?) enjoyed reading this little ray of sunshine of mine. If you enjoyed it, then please come back for the second half of this two-part thriller: “10 Reasons Why I Hate Blogging – Part 2” (innovative name).

I hope this too will help to spread the message that blogging is not for everyone!

Written by Sam Massey, DSCallards

How About Deleting All Your Business Reports Now?

Well how about it?  What would happen if you deleted every business report you had in the company and started again?

I can feel the pain from here. But if your disaster recovery did not work after a hard drive failure this is where you would be.

And once that “all is lost” feeling faded into the acceptance of the “ok, let’s start again” feeling, then this is now a spring cleaning project. Let’s be honest you had been wanting to do this for ages haven’t you?

It’s a bit like living in a house that is not quite right, and you secretly wish it would burn down so that you could claim the insurance and rebuild. The bedroom would be bigger, the kitchen could extend out the back, you could move the bathroom a little further down the hall, etc…

But this only works if you have a couple of hours warning to get out all of the important things like the photo albums and the memorabilia, the guitars, the kids toys and the jewellery, etc… Oh, and the wife, kids and dogs!

But thinking back when we had all of these reports, people were either never using half of them, or they would not trust their results because somebody else had a report that had different figures.

So all this work over the years was an asset, that to be honest was costing a fortune to maintain, and secretly nobody trusted … or even used. At the same time, when figures did not match across even the simplest sales reports … managers were embarrassed in meetings, and in turn their figures were not trusted.
Heated discussions on why my spreadsheet was right and yours must be wrong.

Any boat owner will tell you that the second best day of their boat-owning life is the day they buy their boat.

The best day is the day they sell it. Boats just take a lot of management. You have to clean them, fuel them, fix them, and use them, and if you have a boat in the driveway that you never use, you just feel guilty. Guilty for not using something you spend so much time and money on.

I recently read the story of a CEO of a large Fortune 500 company asking his IT department to delete every report in their report repository except 12 reports, which he listed on a small piece of paper.

I was going to ask why, but there was a bit of me totally understood.  The actual reason was because he had been relying on historic reports that were proved to be incorrect when analysed in a board meeting.  At first he was embarrassed … and then he was angry.

So he went back to the IT department and asked for this to be resolved.

“We will need to hire 3-4 more report designers to handle that request”, came the reply. When he asked why, he was told that the company had over 800 reports in their reporting repository and it would take that number of people to check them through.

So he requested that all of the reports be archived off (as near to being deleted as was possible). Then the 12 reports he used on a daily basis were to be rebuilt.  Finally for the next 3 months, any reports that were requested were to be built afresh and fully documented. Figures were to be matched across all the reports and signed off before they were turned live to the business.

Ok, this took 3 months to get to 45 accurate, tested and trusted business reports. But it indicated that the other 750 were now not needed and as suspected by the CEO, were at best not really wanted … and at worst, were bringing back incorrect information.

At the same time the CEO introduced a new policy.  They put the report stakeholder's name in the footer of every report, and a full description of the reports purpose and where the data came from. That way, if someone requested a report they thought could be satisfied with an existing report, they knew who to call to consolidate the new report.

With an executive mandate from the CEO, the other executives were more willing to compromise and use and trust the same reports.

So feel free again … start deleting. Enjoy the freedom and liberation it gives you!

Now spend time trusting the information and use this to make business decisions that can truly make change …

For more information visit www.crystalreports.co.uk

Written by Ray Kemp, Technical Director, DSCallards

Building Blocks – Objective C

You can only work for so in the world of Objective C before you become exposed to the concept of ‘Blocks’. A very powerful tool in the arsenal of any developer they can drastically increase the amount of code reuse that a developer can leverage in their program. Code reuse is of course a great thing as tried and trusted code can be relied upon to perform correctly and will need the minimum amount of testing whereas duplicated code tends to, as a former colleague of mine used to remark,  lead to the worst kind of inheritance; that is :-  ‘clipboard inheritance’ responsible for many a brainteasing bug. I have also seen it reported that Blocks effectively give you ‘lambda’ in Objective C which as a heavyweight user of LINQ in my preferred language C# I must admit makes me feel a little easier about taking on more Objective C work. They are however a little bit tricky to get your head around when you first come to dealing with them.

Click here to see full blog article.
Written by Conrad Rowlands, Senior Systems Architect and Developer, DSCallards

Using Visual Studio Templates to Scaffold ‘Framework Stacks’

Any applications developer worth their salt will, even when they have downtime, look for new ways to make themselves more productive whilst retaining their quality levels. One such way that we do this is to have ‘Application Frameworks’, common blocks of code and infrastucture that we use on every project. If we were to write this infrastructure and wiring on every project it would just add to the cost and to the drama.

Click here to see full blog article.

Written by Conrad Rowlands, Senior Systems Architect and Developer, DSCallards

iPad Orientation Issues

On a recent i-Pad development project using Phonegap and the Ionic framework we ran into issues with the orientation seemingly being set to portrait only. We had been labouring under the somewhat misplaced apprehension that changes in orientation were just supported with no code being necessary.

Upon looking further into this and seeing what other peoples experiences were there seemed to be a lot of people writing handlers to display when the orientation changed such as this:-

Click here for more.

Written by Conrad Rowlands, Senior Systems Architect and Developer, DSCallards

On the Beach with Yellowfin

Tom Clifford – IT Manager: 1^st June 2014

So it was about time we had a holiday, we had not been away for over a year and both myself and the wife had been working really hard on IT projects for our businesses.

My project needed something else, something that would gather everything together but I was not sure what and I needed time to think, time to go over how this would work when we delivered it to the board, the management and our users.

So this morning I’m off down the travel agent to see what they have available.

Ray Kemp – Technical Director at DSCallards: 1^st June 2014

My dev guys have been working for a couple of weeks now trying to bring together as much social media information as possible. We are attempting to connect twitter, facebook, web page hits, linked in; basically as much as information we can get to form a profile of our customer … those who follow us and those who talk about us here at DSCallards.

This is not wasted time as we have a customer who is looking for this requirement and we wanted to understand the complexities. As we moved from one social network to another it was good to see that they were all using oAuth as a standard interface … a little hassle with facebook but the information started streaming in.

Tom Clifford – IT Manager: 1^st June 2014

Well that was lucky … we had always fancied Mauritius, and the local agent had a last minute deal, better than I could find on the web so we are booked for next week. The photo of the resort accommodation looks breath taking.

The wife is over the moon, decided to travel light … we needed very little. I checked with the resort and they have free public wifi, so the only technology I am taking is my tablet … perhaps if I can clear my mind of all the other project work going on I can investigate how we are going to eventually present this information to the board, the management and the users.

Ray Kemp – Technical Director at DSCallards: 6^th June 2014

It’s taken a week to collate this information, and eventually had to use an open source database that holds the information in document objects rather than the normal row/column format of standard server based databases. This was because each social source had a totally different structure and needed storing and indexing in a specific form.

Once stored, we watched the data updating real-time by the web services our amazing development team had built. We then needed to find relationships between these sets of data.

Page hits, location data, date and time, locales, related word searches within twitter … all this information was initially confusing. But within a day I had our BI team finding relationships … and things started to make sense. They have such a set of skills, just like the development team … you ask for it and it just starts to happen!

I decided to use Yellowfin … we are the UK partner for Yellowfin and I knew we could get to the information quickly once the development and BI teams had done their work. This stuff is so easy to use and so powerful.

Having been down to Melbourne recently talking over the new features in Yellowfin … I knew this would not take long.

Tom Clifford – IT Manager: 7^th June 2014

Well we made it. The flight was not too bad and once we landed we needed to be shuttled across to the resort on the island. It feels like the rest of the world is a million miles away.

The beach is stunning … we went in snorkelling this morning and the water is so clear you could see forever.

So the wife has gone off to be pampered with a massage and some beauty treatment this afternoon. About time I had some time to myself to see if I can find a solution to the data retrieval within this BI project.

Let’s get the tablet out whilst I’m on the beach and see what there is?

BI tools and dashboards, that’s what we need something that looks good to the users and can help management see things out of the data that you cannot normally track.

Hey, this looks good … Yellowfin. Let’s download some of these white papers. Gartner have some great feedback, they are in the quadrant which is important to get buy in from the business. It looks like these guys are learning from others and getting it right.

Nice website, looks like DSCallards are the guys I need to call when I get back. Have tweeted my guys back in the UK to check this out but while I have some peace and quiet, let me go through the website …

Ray Kemp – Technical Director at DSCallards: 7^th June 2014

Ok, the data has started streaming in now and Yellowfin is working really well. The data views were easy to put together. So let’s see who is checking out our website and tweeting our details around the same time.

“Hey guys … come and look at this. There is somebody in Mauritius with a tablet browsing our website and he is tweeting about Yellowfin. He’s got to be on the beach!”

Note: These are actual stats taken at 12:25am on 7^th June 2014. Thanks to Yellowfin and our stunning development and BI teams here at DSCallards ... I know nothing is impossible.

Written by:  Ray Kemp, Technical Director, DSCallards

Reasons to Use SAP Crystal Reports SQL Expression Fields

Overview

These objects provide an excellent alternative to using a formula in Crystal Reports. The main advantage being that the processing of an SQL Expression field is performed at the database server side and not on the client machine.

The SQL Expression functionality will vary depending on the database you connect to. One useful thing to know is that Crystal Reports supports any SQL function for the database you connect to, even if the function name is not visible in the Functions area of the SQL Expression Editor.

Figure 1 below shows an example of the Microsoft SQL Server DATEADD function.

(Figure 1)

Comparisons between a Formula and an SQL Expression field?

The SQL Expression Editor bears many resemblances to the Formula Editor. As a report designer, you will need to know the SQL Syntax to successfully utilise SQL Expressions in your Crystal Reports. Let’s take a look at the example in Figure 1. The DateAdd formula syntax is almost identical. Figure 2 below shows the formula syntax.

(Figure 2)

Report Optimisation

This is the main reason for using SQL Expressions over formulas. SQL Expression fields allow a user defined calculation in a Crystal Report to be processed on the database server. This means that the column is derived and sent to the client machine. Compare this with a formula that is wholly processed on the client machine. Figure 3 shows the SQL syntax for the DATEADD function being derived on the database server.

(Figure 3)

A valid equivalent to ‘if-then-else’

Using the SQL CASE statement is a very useful alternative to the very popular ‘if-then-else’ Crystal syntax. Figure 4 shows a simple CASE statement.

(Figure 4)

Pre-defined SQL Expression functions

These are set out in an almost identical way to the formula functions in the formula editor. Figure 5 shows the main function area.

(Figure 5)

Using a pre-defined SQL Expression function is easier because you can simply select the function, field and arguments without typing the SQL. Figure 6 shows an example of the pre-defined MONTHNAME function.

(Figure 6)

And as you would expect, it is derived on the database server. Figure 7 shows the SQL syntax below.

(Figure 7)

Written by Steve Chapman, Senior BI Consultant, DSCallards